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Summary rel 2009-No. 072 

How state education agencies in 
the Northeast and Islands Region 
support data-driven decisionmaking 
In districts and schools 



The report examines the initiatives of 
state education agencies in the Northeast 
and Islands Region to support data- 
driven decisionmaking in districts and 
schools and describes the service provid- 
ers hired to support this work. Identi- 
fying four components of data-driven 
decisionmaking initiatives, it finds that 
not all initiatives include all four. 

Data-driven decisionmaking is receiving in- 
creasing attention from the education commu- 
nity because of federal and state accountability 
requirements; the enhanced capacity of states 
and school districts to collect, manage, and 
distribute data; and a better understanding of 
the importance of data-driven decisionmaking 
in improving instruction and student achieve- 
ment. The report responds to a request by the 
Rhode Island Department of Elementary and 
Secondary Education for information about 
how other state education agencies in the 
Northeast and Islands Region support districts 
and schools in effectively using data to inform 
a range of education decisions. 

Two research questions guided this study: 

• What state education agency initiatives 
support data-driven decisionmaking by 



districts and schools in the Northeast and 
Islands Region? What are their similarities 
and differences? 

• What service providers do the state 
education agencies work with to support 
educators in using data to inform educa- 
tion decisions, and what services do they 
provide? What are their similarities and 
differences? 

The descriptions of initiatives by state educa- 
tion agencies in the Northeast and Islands 
Region to support data-driven decisionmaking 
in districts and schools are based on exami- 
nations of state education agency web sites, 
interviews with agency officials, and a review 
of the documents they provided. State educa- 
tion agency officials also identified the service 
providers they have hired to support this 
work. Three providers were selected for more 
in-depth profiles. Descriptions of the three 
service providers are based on interviews with 
service provider staff, observations of profes- 
sional development activities, and additional 
materials provided by the service providers. 
Appendixes to the report include profiles 
of each state education agency initiative, a 
catalogue of nine service providers identi- 
fied by the state education agency officials. 
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and profiles of three service providers. This 
study has several limitations, including the 
small number of respondents interviewed, the 
focus on state education agency data-driven 
decisionmaking supports for districts and 
schools rather than on district and school im- 
plementation of data-driven decisionmaking, 
and a lack of information about the decisions 
made by state education agency officials. 

Analysis of the state education agency data- 
driven decisionmaking initiatives across 
the region revealed that state agencies have 
implemented one or more of four components 
to support data use by schools and districts: 

• Centralized data system/warehouse. A cen- 
tralized data system/warehouse combines 
data from multiple sources into a central- 
ized repository. Data can include a range 
of evidence, such as classroom assessment 
data, school-level information on students 
and staff, demographic data, and state test 
scores. 

• Tools for data analysis and reporting. Data 
tools allow users to collect, organize, and 
analyze data for use in decisionmaking. 

• Training on data systems/warehouses and 
tools. Training helps educators learn to 
effectively and efficiently use the data 
analysis tools to better understand the 
available data. 

• Professional development in using data 
for decisionmaking. Teachers and admin- 
istrators require extensive professional 
development to build their expertise in 
identifying and analyzing relevant data 
and adjusting instructional practices 



and school processes in response to such 
data. 

Each state education agency’s initiatives were 
reviewed within the framework of these 
components. Only New Hampshire appears to 
provide all four components to every school in 
the state. Maine and Vermont provide all four 
components but target only specific schools. 
New York provides a data system, data tools, 
and professional development, but as separate 
initiatives with little overlap. Massachusetts, 
Rhode Island, and the Virgin Islands are creat- 
ing a longitudinal data system and data tools 
and developing training in their use. Connect- 
icut focuses solely on the process of using data 
for decisionmaking and targets that support to 
low-performing districts. 

Despite a dearth of comprehensive initiatives, 
state education agency officials across the re- 
gion mentioned the importance of providing a 
range of data-driven decisionmaking supports 
to schools. But they noted that limited fund- 
ing and a lack of capacity force them to make 
choices about which components to provide. 

Analysis of the four components across the 
state education agency initiatives revealed that 
implementation is affected in part by available 
funding and capacity: 

• Funding. State education agency data- 
driven decisionmaking initiatives were 
shaped by the funding available. Exter- 
nal funds (such as Title I) may restrict 
who can receive services or prescribe 
the types of services that can be offered. 
State funding must be stretched to cover 
a range of education services, and sup- 
port for data-driven decisionmaking must 
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compete with teacher salaries and hook 
and supply purchases. 

• Capacity. State education agency staff may 
lack the capacity to support their data- 
driven decisionmaking initiatives because 
of time or expertise. To help provide 
needed services to schools and districts, 
state education agencies have contracted 
with external service providers. 

State education agency officials identified the 
following providers of support for data-driven 
decisionmaking initiatives in the region: Cen- 
ter for Assessment, Cognos, the Connecticut 
Regional Educational Service Centers Alliance 
(RESC Alliance), ESP Solutions Group, Mea- 
sured Progress, the New York Board of Coop- 
erative Educational Services (BOCES), Pearson 
School Systems, Performance Pathways, and 
TetraData. Three of these nine service provid- 
ers were selected for in-depth profiles: the 
Connecticut RESC Alliance, Measured Prog- 
ress, and Performance Pathways. 

These three service providers assist state 
education agencies in implementing the four 
key data-driven decisionmaking components 
identified in this study: 

• Centralized data system/warehouse. Perfor- 
mance Pathways and Measured Progress 
support state education agencies in creating 
a central repository for storing data. 

• Tools for data analysis and reporting. 

All three service providers use various 
tools to support data analysis and report- 
ing, ranging from paper and pencil to 
computer-based spreadsheet software and 
online programs capable of generating 



customized reports from the centralized 
data system. 

• Training in data systems/warehouses and 
tools. Performance Pathways and Mea- 
sured Progress not only support the cre- 
ation of a data warehouse system but also 
provide training in using the system and 
its tools. To reach a broader audience, both 
providers use a “train the trainer” model, 
training groups of educators who are then 
responsible for training colleagues. 

• Professional development in using data for 
decisionmaking. All three providers offer 
some professional development in using 
data to guide decisionmaking. Offerings 
vary, depending on the organization’s 
focus on creating a data-driven culture, 
using assessment data, or developing 
software tools. Providers work with teams 
from districts or schools who are expected 
to train their colleagues in the process. 

This study outlines several considerations 
for education decisionmakers and research- 
ers on the potential benefits of implement- 
ing additional components of a data-driven 
decisionmaking system, sources of funding, 
and strategies to enhance their capacity to 
support teachers and administrators. Ideas 
are proposed for further research, including 
examining how state education agencies scale 
up their data-driven decisionmaking initia- 
tives; exploring how state education agencies, 
schools, and districts implement data-driven 
decisionmaking; and analyzing the impacts of 
data-driven decisionmaking on student and 
school outcomes. 

May 2009 
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The report 
examines the 
initiatives of state 
education agencies 
in the Northeast 
and Islands 
Region to support 
data-driven 
decisionmaking 
in districts and 
schools and 
describes the 
service providers 
hired to support 
this work. 
Identifying four 
components 
of data-driven 
decisionmaking 
initiatives, it 
finds that not 
all initiatives 
include all four. 



WHY THIS STUDY? 

Use of data is recognized as a major component of 
school improvement. No Child Left Behind (NCLB) 
legislation stipulates that low-performing Title I 
schools use data to support school improvement 
plans submitted to the state education agency 
(NCLB section 1116). Such data could include test 
scores, classroom assessments, attendance and 
discipline records, school or district staffing pat- 
terns (teachers’ and principals’ years of experience, 
areas of expertise, and levels of education), and 
classroom information gathered systematically by 
teachers (Wayman 2005). The increased pressure 
for data-driven decisionmaking comes not only 
from external accountability requirements, such as 
the NCLB Act and state legislation, but also from 
an increased understanding that school improve- 
ment efforts benefit from systematically gathering 
and analyzing information about the school, its 
students, and the school’s capacity to meet stu- 
dents’ needs (Finnigan, O’Day, and Wakelyn 2003; 
Marsh et al. 2005; Ikemoto and Marsh 2007). 

Data-driven decisionmaking is neither new nor 
limited to education decisionmakers (see box 1 for a 
description of data-driven decisionmaking and defi- 
nitions of key terms). Analyzing data to assess or- 
ganizational effectiveness and to craft improvement 
strategies has long been important for managers in 
the public and private sectors (Feigenbaum 1951; 
Deming 1982; Barzelay 1992). Although educators 
have a long tradition of using test scores and other 
information to shape instructional decisions, until 
recently the use of data to inform education deci- 
sions has not been systematic (Abelman et al. 1999). 
With external accountability pressures and the 
availability of new technology to access data, there 
is now a push by state education agencies for more 
uniform and rigorous use of data to improve in- 
struction outcomes (see, for example, Abbott 2008; 
New Hampshire Department of Education 2006). 



Importance to the region 

Officials at the Rhode Island Department of 
Elementary and Secondary Education expressed 
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BOX 1 

What is data-driven 
decisionmaking? 

Key terms related to data-driven 
decisionmaking include the following 
(for more detail see appendix A): 

Data-driven decisionmaking in edu- 
cation refers to “teachers, principals, 
and administrators systematically 
collecting and analyzing various 
types of data ... to guide a range of 
decisions to help improve the success 
of students and schools” (Marsh, 
Pane, and Hamilton 2006, p. 1). It can 
include the relatively simple process 
of disaggregating state test scores to 
identify and support struggling stu- 
dents (Ikemoto and Marsh 2007) and 
the more complex process in which 
faculty develop a school environment 



that supports the use of evidence to 
“reshape the central practices and 
cultures of their schools to react 
intentionally to the new kinds of 
data” (Halverson et al. 2007, p. 4). 
Strategies include forming data teams 
to guide the data-driven decision- 
making process (Love et al. 2008) 
and implementing new instruction 
approaches and the associated profes- 
sional development (Moody, Russo, 
and Casey 2005). 

Culture of inquiry is a type of data- 
driven decisionmaking in which 
faculty create an organizational 
culture focused on using data and 
other evidence to shape instructional 
practices. 

Data collected about students, teach- 
ers, and their schools and districts 



can include scores from large-scale 
assessments, classroom assessments, 
attendance and discipline records, 
and statistics on school or district 
staffing patterns (such as teachers’ 
and principals’ years of experi- 
ence, areas of expertise, and level of 
education). 

State data-driven decisionmaking 
initiatives are initiatives that have 
been conceptualized at the state level 
to support data use by local educa- 
tors; have a set of comprehensive 
goals, objectives, and purposes; and 
have designated resources (finan- 
cial and personnel) to support their 
development and implementation. In 
addition, state education agency lead- 
ers must communicate their policies 
to districts in the state and offer them 
support. 



interest in research that describes how state educa- 
tion agencies can support data-driven decision- 
making in districts and schools. The department 
is designing a comprehensive data system to 
provide educators — from classroom teachers to 
district and state administrators — with access to 
data. This effort prompted Rhode Island Depart- 
ment of Elementary and Secondary Education 
officials to request the Regional Educational 
Laboratory Northeast and Islands to profile how 
other jurisdictions in the region support system- 
atic data-driven decisionmaking in districts and 
schools. Officials at state education agencies in 
Massachusetts, New Hampshire, and New York 
also expressed interest in the project. 

Extensive research on data-driven decisionmaking 
at the district and school levels reveals the variety 
of practices teachers and administrators used 
to support both classroom practices and school 
improvement efforts (see appendix C for a brief 
review of the literature). The research findings are 
also clear that educators need external support to 



develop their capacity to conduct these activities. 
But there is little research on how state education 
agencies and external service providers support 
these efforts— the focus of this report. 



Study questions 

Two research questions guided this study: 

• What state education agency initiatives sup- 
port data-driven decisionmaking by districts 
and schools in the Northeast and Islands 
Region? What are their similarities and 
differences? 

• What service providers do the state education 
agencies work with to support educators in 
using data to inform education decisions, and 
what services do they provide? What are their 
similarities and differences? 

The first research question concerns state educa- 
tion agency initiatives to develop the capacity of 
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district and school leaders to use data to adjust 
instruction and organize schools to improve stu- 
dent achievement (see box 1 for definitions of key 
terms). The second research question examines 
service providers in the region and how they sup- 
port data-driven decisionmaking in districts and 
schools. State education agencies contract external 
service providers to support data-driven decision- 
making initiatives by providing technology tools 
and professional development. The responses to 
these research questions will be useful to state 
leaders who are designing or implementing initia- 
tives for building the capacity of educators to 
conduct data-driven practices at the school and 
district levels and to district administrators who 
sometimes contract with service providers to sup- 
port data-driven decisionmaking. 

To answer these questions, the project team con- 
ducted a qualitative study of data-driven decision- 
making initiatives in the Northeast and Islands 
Region (see box 2 and appendix B for details on 
the study methods). The report also provides an 
overview of the literature on data-driven decision- 
making in schools and districts and the external 
supports for those efforts (appendix C), case pro- 
files of state education agency initiatives (appendix 
D), a catalogue of the nine providers contracted 
by state education agencies to support data-driven 
decisionmaking in districts and schools in their 
jurisdiction (appendix E), and case profiles of 
three service providers (appendix F). 

The state education agencies and service provid- 
ers in the Northeast and Islands Region profiled 
in this report offer a range of examples of how to 
support data-driven decisionmaking at the district 
and school levels. While ideally state education 
agencies could provide a comprehensive range of 
supports for data-driven decisionmaking, limited 
resources often force tradeoffs between providing 
data and tools and fostering a climate that encour- 
ages a process of data-driven decisionmaking. The 
examples in the report are intended to expand the 
dialogue on how state education agencies can sup- 
port data-driven decisionmaking and help practi- 
tioners better meet the needs of their students. 



WHAT STATE EDUCATION AGENCY INITIATIVES 
SUPPORT DATA-DRIVEN DECISIONMAKING 
BY DISTRICTS AND SCHOOLS? 

The descriptions in this section provide a snap- 
shot of the data-driven decisionmaking initiatives 
and their context in Connecticut, Maine, Mas- 
sachusetts, New Hampshire, New York, Rhode 
Island, Vermont, and the Virgin Islands. Table 5 
at the end of this report compares the major 
elements of the initiative in each jurisdiction, 
including targeted groups, data collected, data 
analysis tools, and service providers working with 
the state education agencies. A full description 
of each state education agency initiative is in ap- 
pendix D. 



State education agency data-driven decisionmaking 
initiatives by state and jurisdiction 



The goal of each state education agency’s work 
with districts and schools on data-driven decision- 
making is to help local educators use data to 
improve student outcomes. Despite this common 
mission, individual agencies in the Northeast 
and Islands Region support data-driven decision- 
making in distinct ways. This section describes 
each state’s approach. 



Connecticut. Connecticut’s data-driven decision- 
making initiative emphasizes improving the 
outcomes of low-performing schools. Under the 
Connecticut Accountability for Learning Initiative 
of the State Department of Education, regional 
educational service centers train school teams 
in low-performing districts to analyze data to 
improve student performance. Classroom and 

school-level data are 

examined, including 
state assessment scores 
from the Connecticut 
Academic Performance 
Test and Connecticut 
Mastery Test, along with 
data on student behav- 
ior and school climate. 

Professional development 



The goal of each state 
education agency's 
work with districts and 
schools on data-driven 
decisionmaking is to 
help local educators 
use data to improve 
student outcomes 
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BOX 2 

Study methods 

Study sample and analysis. The 
researchers used publicly available 
information to identify statewide 
data-driven decisionmaking initia- 
tives, key respondents, and service 
providers; supplemental documents 
provided by respondents from state 
education agencies and service pro- 
viders; semistructured, open-ended 
interviews with key respondents from 
each state education agency and each 
service provider; and observations of 
professional development activities. 
Data collection was iterative, with 
the state education agency initiatives 
informing the selection of service 
providers. 

Interviews were conducted with 
a state education agency official 
with a key role in the data-driven 
decisionmaking initiatives in each 
of the eight state education agencies 
included in the study (Connecticut, 
Maine, Massachusetts, New Hamp- 
shire, New York, Rhode Island, Ver- 
mont, and the Virgin Islands;^ two 
officials were interviewed in Massa- 
chusetts). Each respondent was asked 
to identify at least one service pro- 
vider supporting that state education 
agency’s data-driven decisionmaking 



initiatives and the person responsible 
for implementing the state contract 
(appendix B provides more details on 
the selection criteria). Three service 
providers from the list of nine identi- 
fied providers were selected for more 
in-depth profiles: Measured Prog- 
ress, Performance Pathways, and the 
Connecticut Regional Educational 
Service Centers Alliance (see appen- 
dix F). They were selected because 
they serve multiple jurisdictions in 
the region, have been in existence 
for two or more years, have provided 
data-driven decisionmaking ser- 
vices for at least two years, and offer 
professional development services 
in using data to inform teaching and 
learning. 

Interviews were recorded and tran- 
scribed. Transcripts and observation 
notes were coded using the qualita- 
tive data analysis software ATLAS-ti. 
The coding output was analyzed 
alongside documents to triangulate 
the evidence about state education 
agency policies and service provider 
efforts. Researchers used these data 
to develop state case profiles and ser- 
vice provider profiles. The completed 
profiles facilitated systematic analysis 
of cross-state education agency and 
cross-provider issues. The research 
team met regularly to share case 



profiles and to develop cross-case 
themes. 

Limitations of the study. The primary 
limitation of this study is the small 
number of respondents interviewed. 
The knowledge and opinions of one 
state education agency official from 
each jurisdiction may not represent 
the knowledge or perspectives of all 
state education agency staff. In addi- 
tion, information in the three service 
provider profiles cannot be general- 
ized to represent the six providers 
that were not profiled. Also, the study 
focuses on the role of state education 
agencies and does not explore in any 
depth the role of districts in sup- 
porting data-driven decisionmaking 
initiatives. Finally, the study does not 
examine the effectiveness of any of 
these initiatives or their impacts on 
schools. 

Note 

1. The plan originally included all nine state 

education agencies in the region. Despite 
attempts over several months, the re- 
search team was unahle to coordinate an 
interview with an appropriate individual 
from the Puerto Rico Department of 
Education. As the study moved into the 
second phase of the research, focusing on 
service providers, a decision was made 
to proceed without information from 
Puerto Rico because of time constraints. 



in data-driven decisionmaking for teachers and 
school administrators focuses on creating data 
teams and training them to build and sustain a 
five-step process of embedding data and evidence 
into their decisionmaking processes. No technol- 
ogy is used during the training; data are typically 
on paper. Certification training, intended for 
educators who want to create a schoolwide culture 
of inquiry through facilitation of ongoing profes- 
sional development at their schools, is provided 



by the Leadership and Learning Center, a partner 
organization. 

The Connecticut State Department of Educa- 
tion does not have a statewide data warehouse to 
store student assessment data, but officials are 
considering one. The regional educational service 
centers had provided a statewide warehouse to 
districts through a vendor, TetraData, but that 
relationship ended in June 2008. Connecticut 
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received a longitudinal data system grant from 
the U.S. Department of Education in 2005. Ac- 
cording to the U.S. Department of Education web 
site, the Connecticut State Department of Educa- 
tion is using the grant to enhance its technology 
infrastructure to support data collection, but 
not to support data-driven decisionmaking in 
districts and schools. 

Maine. Although Maine lacks a formal data-driven 
decisionmaking initiative, state education leaders 
have long recognized the need to assist local edu- 
cators in developing a culture of inquiry in schools 
to support the use of evidence in decisionmaking. 
Since the early years of the Maine Educational 
Assessment (first administered in 1984), the 
Maine Department of Education and Measured 
Progress, a service provider focusing on assess- 
ment development, have provided workshops to 
district and school administrators to support them 
in interpreting the state assessment reports. In 
addition, in the 1990s and the early 2000s the state 
accountability system included local performance 
assessments.' In recent years the workshops also 
have trained educators on an online data tool, 
designed by Measured Progress, that allows users 
to customize the Maine Educational Assessment 
reports. Administrators are then expected to share 
this training with their colleagues. 

Maine Department of Education staff provide 
follow-up support for school teams — to conduct 
item analysis and to discuss how to identify stu- 
dent needs and strengthen instruction based on 
state assessment evidence— but very few personnel 
are available to serve the state, which is as large as 
the rest of New England combined. Department 
staff support data-driven decisionmaking within 
specific programs (such as Title I, Reading First, 
and special education), but limited state funds 
mean that these services are available only to a 
targeted set of schools. Maine is using a federal 
longitudinal data grant to develop a new data 
system that will give all schools access to an online 
longitudinal data system that will integrate state 
assessment scores with student information and 
staffing data. 



Although Maine lacks 
a formal data-driven 
decisionmaking initiative, 
state education leaders 
have long recognized 
the need to assist local 
educators in developing 
a culture of inquiry 
in schools to support 
the use of evidence 
in decisionmaking 



Massachusetts. The 
Massachusetts Depart- 
ment of Elementary and 
Secondary Education has 
a data-driven decision- 
making initiative that 
focuses on creating a 
virtual data warehouse 
and sophisticated tools 
to help educators analyze 
the data. In 2001 the 
department created a 
centralized system to 

collect student background information and link 
it to Massachusetts Comprehensive Assessment 
Systems data. For the past three years the depart- 
ment has been developing and pilot testing its 
virtual Educational Data Warehouse initiative, 
which will give people at all levels (state, district, 
and school) access to a variety of state-maintained 
data (such as state assessment scores and student 
and educator information) and data analysis and 
reporting tools. Districts will have the opportunity 
to add their own data, including course grades and 
attendance or discipline data. 



The virtual Educational Data Warehouse uses 
software developed and managed by Cognos 
Corporation. As of May 2008, 76 districts (of 
388) were piloting the data warehouse. The pilot 
includes training on the data tools and profes- 
sional development support for implementing a 
process of data-driven decisionmaking. Once the 
pilot ends and the initiative is available statewide, 
districts will need to provide training on the data 
analysis tools and professional development from 
their own budget. 



New Hampshire. In 2007 New Hampshire unveiled 
Performance Tracker, a suite of software developed 
by Performance Pathways to support the state’s 
formal data-driven decisionmaking initiative. 
Follow the Child. Follow the Child focuses on 
measuring growth in the personal, social, physical, 
and academic facets of each student’s life and on 
defining the support systems students need to suc- 
ceed. The intention is to ensure that personalized 
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learning and student well-being are at the center of 
district and school policies. 

Along with storing and analyzing student and 
educator data, Performance Tracker can link 
curriculum, assessments, and standards and help 
educators create and store local assessments. New 
Hampshire, like Connecticut and Maine, is the 
recipient of a grant from the U.S. Department of 
Education to improve its longitudinal data system. 
The federal support has allowed New Hampshire 
to waive the subscription fee for districts to the 
state’s data warehouse. New Hampshire also is 
using its federal grant to provide professional 
development for educators at all levels — from 
preschool to postsecondary — to make better use of 
multiple kinds of data to support the learning and 
development of each child and to foster a climate 
that supports the use of data and evidence to guide 
instruction. The New Hampshire Department of 
Education appears to have the most comprehen- 
sive data-driven decisionmaking initiative in the 
Northeast and Islands Region. 

New York. Although the New York State Educa- 
tion Department does not have a formal statewide 
data-driven decisionmaking policy for districts 
and schools, its support for low-performing 
schools encourages data-driven decisionmaking. 

It also encourages state-sponsored and other 
agencies to incorporate supports for data-driven 
decisionmaking into their programs. 



Although the New 
York State Education 
Department does 
not have a formal 
statewide data-driven 
decisionmaking policy 
for districts and schools, 
its support for low- 
performing schools 
encourages data-driven 
decisionmaking. 



New York has a strong history of using data to 

inform education decisions at all 

levels of the system. A network of 
regional school support centers, 
created in 2003, provides profes- 
sional development for teachers in 
low-performing schools in using 
data to drive instructional deci- 
sions. The New York State Testing 
and Accountability Reporting 
Tool (nySTART), introduced in 
2006, gives school leaders access 
to a variety of reports on scores 
on New York’s standardized tests. 



Educators can view and sort data by performance 
level, content strand or standard, test item, and 
subgroup. District and school administrators and 
teachers are expected to use the information to 
inform education decisions. Furthermore, the 
regional Boards of Cooperative Educational Ser- 
vices (BOCES), created in 1948 to provide shared 
education programs and services to New York 
school districts, offer data-driven decisionmaking 
services based on the needs of the regions they 
serve. Many low-performing schools seek BOCES 
services in using data to improve teaching and 
learning. 

Rhode Island. The Rhode Island Department of 
Elementary and Secondary Education supports 
data-driven decisionmaking in the state through 
major investments in a data warehouse system to 
support its longitudinal data system and tools to 
enable districts to use the data to inform instruc- 
tional decisions. State, district, and school educa- 
tors and parents will have access to the online 
consolidated data warehouse, which facilitates 
data collection and provides data analysis tools. In 
2006 the Rhode Island Department of Elementary 
and Secondary Education contracted with Tetra- 
Data and ESP Solutions Group to create a central 
virtual repository to consolidate vast amounts of 
information and data that had been available in 
separate locations (such as New England Common 
Assessment Program data,^ student and financial 
information, and teacher certification). The system 
is being introduced gradually over a two-year 
period. During the roll-out district staff are being 
trained in how to use the components of the new 
data system. 

Vermont. Since the mid-1990s the Vermont De- 
partment of Education has supported educators’ 
use of data to improve instruction and student 
outcomes. However, before passage of the NCLB 
Act, these initiatives were not tied to accountabil- 
ity efforts. Currently, the department has a formal 
data-driven decisionmaking initiative with spe- 
cific requirements as part of its work with low-per- 
forming schools. It also informally provides some 
resources related to data-driven decisionmaking to 
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all schools in the state. Schools in need of improve- 
ment must provide data-based evidence of the 
impact of their school improvement efforts on stu- 
dent learning.^ These schools receive support and 
training for data-driven decisionmaking processes 
through principal learning communities;'* by 2009 
similar professional learning communities will be 
required for all teachers in these schools through 
teacher learning communities. 

In addition to services provided free of charge to 
low-performing schools, districts can subscribe to 
the Vermont Data Warehouse, which has both data 
storage and analysis capabilities. Fees are based 
on a sliding scale, according to district size and 
revenues. The warehouse uses software developed 
by TetraData and is administered by the Vermont 
Data Consortium, a partner agency of the Vermont 
Department of Education. The data warehouse 
provides training on the use of its data tools. 

Virgin Islands. The Virgin Islands Department of 
Education does not have a formal policy to support 
data-driven decisionmaking in its districts and 
schools. A department official noted that the juris- 
diction relies on the provisions in the NCLB Act to 
encourage the use of data in decisionmaking. The 
Virgin Islands Territorial Assessment of Learning 
(VITAL) has been in place since 2004/05; before 
that the jurisdiction had had a five-year morato- 
rium on standardized testing. Since territorywide 
testing resumed in 2004, the assessment has been 
taken by students in grades 5, 7, and 11 in each 
of the past three test administrations. The Virgin 
Islands Department of Education is beginning to 
develop a longitudinal data system to access and 
analyze both assessment data and student infor- 
mation, and the professional development that 
supports the use of the data system is part of a 
contract with Pearson School Systems, the exter- 
nal service provider that created the system. 



Four key components to support data collection and use 

Analysis of state education agency data-driven 
decisionmaking initiatives in the Northeast and 
Islands Region revealed that agencies typically 



have one or more of four 
key components in place 
to support data collection 
and use by schools and 
districts in making edu- 
cation decisions. These 
components provide a 
logical framework for 
presenting findings. 



Analysis of state 
education agency data- 
driven decisionmaking 
initiatives in the 
Northeast and Islands 
Region revealed that 
agencies typically have 
one or more of four key 
components in place to 
support data collection 
and use by schools 
and districts in making 
education decisions 



Centralized data 
system/warehouse. 

A centralized data 
system/warehouse 
combines data from 

multiple sources into a centralized repository. 
Such centralized systems provide a range of 
stakeholders with an integrated view of mul- 
tiple data sources and provide a customized 
interface to access these data (Dembosky et al. 
2005). Data range from classroom assessment 
data to school-level information about stu- 
dents and staff (Gallagher, Means, and Padilla 
2007; Wayman 2005). 



• Tools for data analysis and reporting. Data 

tools are software that allows teachers and ad- 
ministrators to collect, organize, and analyze 
data and use them for decisionmaking (see, 
for example, case studies by Larocque 2007; 
Storandt, Nicholls, and Yusaitis 2007; Chen, 
Heritage, and Lee 2005; Lachat and Smith 
2005; Light et al. 2005; Murnane, Sharkey, and 
Boudett 2005; Mason 2002). These software 
tools can reside on individual personal com- 
puters or be provided online. User-friendly 
software allows easy access to and manipula- 
tion of data to inform decisions about class- 
room practice. 



• Training on data systems/warehouses and 
tools. Educators often need training on 
effective and efficient use of data systems/ 
warehouses and tools (Earl, Katz, and Fullan 
2006; Dembosky et al. 2005). Teachers and 
district and school administrators may receive 
technical training and support through 
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Six of the eight state 
education agencies 
provide data reporting 
and analysis tools to 
local educators, often 
with the support of an 
external service provider 



workshops, user conferences, 
review of annotated score reports, 
or technical assistance sessions led 
by project managers. 



• Professional development in 
using data for decisionmaking. 
Professional development in 
data-driven decisionmaking 
builds educators’ expertise 
and habits in identifying and analyzing rel- 
evant data and adjusting instruction and other 
practices in response to the data (Boudett and 
Steele 2007; Knapp, Copland, and Swinnerton 
2007; Young 2006; City, Kagle, and Teoh 2005; 
Lachat and Smith 2005; Murnane, Sharkey, 
and Boudett 2005; Copland 2003; Feldman and 
Tung 2001). Professional development occurs 
over time and often involves an iterative cycle 
in which teachers examine data, plan for im- 
plementation of targeted strategies, implement 
those strategies, and evaluate their impact 
(Abbott 2008). It often focuses on building 
collaborative teams that use data to examine 
improvement efforts (Love et al. 2008). Such 
professional development is often delivered 
through workshops or ongoing coaching for 
data teams in schools and districts. 



These four components have been implemented 
in varying combinations and sequences and are at 
different stages of implementation in the agen- 
cies studied. Choices about which components to 
implement — and about the depth and breadth of 
implementation — are shaped at least in part by 
the availability of funding and the implementation 
capacity of state education agencies. The agencies 
are extending their capacity to support data use in 
districts and schools by contracting with organiza- 
tions that can provide services for one or more of 
the data-driven decisionmaking components. 



Similarities and differences in data-driven decisionmaking 
initiatives across state education agencies 



four key components of data-driven decision- 
making, focusing on similarities and differences. 
It also looks at how available funding and capac- 
ity for data-driven decisionmaking supports 
shaped the decisions of state education agencies 
on which components to implement, who would 
receive access to these supports, and which 
outside vendors to work with states to provide 
supports. 

State education agencies in the Northeast and 
Islands Region have implemented the four compo- 
nents to different extents and in varying combina- 
tions and sequences (see table 1 for an overview of 
the components). 

Centralized data system/warehouse. The state edu- 
cation agencies in the region offer various systems 
to support data-driven decisionmaking in districts 
and schools. These include consolidated virtual 
data warehouses (Massachusetts, New Hampshire, 
Rhode Island), a data warehouse run by a part- 
ner organization rather than the state education 
agency (Vermont), integrated but more limited 
data systems (Virgin Islands), and separate reposi- 
tories for different types of data (New York and 
Maine). Maine is developing a consolidated data 
warehouse that should be available to all schools 
within three years. Connecticut plans to develop 
a statewide data warehouse, but currently data- 
driven decisionmaking relies on data collected by 
districts and at schools. 

Tools for data analysis and reporting. Seven of the 
eight state education agencies (all but Connecticut) 
provide data reporting and analysis tools to local 
educators, often with the support of an external 
service provider. In Vermont a partner agency 
provides data tools, which are available to districts 
for a subscription fee. Massachusetts, New Hamp- 
shire, Rhode Island, and the Virgin Islands offer 
software that facilitates analysis of assessment and 
other data when used in conjunction with their 
data warehouses. 



This section looks at the extent to which state The Massachusetts Department of Elementary and 

education agencies in the region implement the Secondary Education has purchased a license for 
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TABLE 1 

Components of state education agency initiatives in the Northeast and Islands Region, 2007/08 



Jurisdiction 


Centralized data system/warehouse 

Consolidated Separate 

data system/ data system/ 

warehouse warehouses 


Tools for data 
analysis and 
reporting 


Training in 
data systems/ 
warehouses 
and data tools 


Professional 
development in 
using data for 
decisionmaking 


Connecticut 










/ 


Maine 




/ 


/ 


/ 




Massachusetts 


✓ 




✓ 






New Hampshire 


✓ 




/ 


/ 


/ 


New York 




/ 


/ 




/ 


Rhode Island 


/ 




/ 


/ 




Vermont 










/ 


Virgin Islands 




/ 


/ 


/ 





a. A limited number of Maine Department of Education staff are available to support data-driven decisionmaking at the school level. 

b. State education agency funding is available only for pilot districts. Available statewide through districts' funds. 

c. These services are provided by a partner organization for a subscription fee. 

Note: Puerto Rico is not included because the study team was unable to set up interviews with key respondents from the Puerto Rico Department of Educa- 
tion, despite repeated attempts over several months. 

Source.- Authors' compilation based on interviews and documents provided by state education agency officials and information available on state education 
agency websites. 



data warehouse software that will house all the 
state-maintained data and will eventually allow 
school districts to load and analyze their own data. 
All New Hampshire districts use Performance 
Tracker, the state-adopted data warehouse and 
analysis tool that is used in conjunction with the 
state’s data warehouse. The Rhode Island De- 
partment of Elementary and Secondary Educa- 
tion is creating a data warehouse to hold all of 
its education-related data. It will include a data 
analyzer available at all levels of the state educa- 
tion system. The Virgin Islands Department of 
Education contracts with Pearson School Systems, 
the vendor that created its system, to access and 
analyze state assessment data and student infor- 
mation. Maine and New York have stand-alone 
web-based tools that allow educators to conduct 
analyses and create reports using assessment and 
other types of data. Maine works with Measured 
Progress to offer a web portal that enables district 
staff and school faculty to generate customized 
score reports from Maine Educational Assess- 
ment data. New York’s nySTART tool gives school 
leaders access to a variety of reports on students’ 
standardized test scores. 



Training in data systems/warehouses and data 
tools. Six of the eight jurisdictions (Maine, Massa- 
chusetts, Rhode Island, New Hampshire, Vermont, 
and the Virgin Islands) provide training on use 
of the data warehouses and data analysis tools. 

In Vermont the training is provided by a partner 
organization. Maine provides workshops on the 
online data tool, designed by Measured Progress, 
which allows users to customize Maine Educa- 
tional Assessment reports. Districts that have been 
piloting the Massachusetts data warehouse have 
received training on uploading assessment data to 
the warehouse and on the analysis tools. District 
staff in Rhode Island are being trained to use their 
new system. Performance Pathways provides New 
Hampshire educators with training in their ware- 
house data analysis and reporting tools. As part 
of the contract with Pearson School Systems, the 
Virgin Islands Department of Education provides 
professional development in the use of its data sys- 
tem to access and analyze both student assessment 
data and student information. 

Professional development in using data for 
decisionmaking. Six jurisdictions (Connecticut, 
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Maine, Massachusetts, New Hampshire, New 
York, and Vermont) provide professional develop- 
ment in using data for education decisionmaking. 
Connecticut provides this training through its 
low-performing districts initiative, Connecticut 
Accountability for Learning Initiative. Profes- 
sional development for teachers and school 
administrators focuses on training data teams to 
build and sustain a five-step process of embed- 
ding data and evidence into decisionmaking 
processes. The Maine Department of Education 
has staff available to facilitate item analysis with 
school teams and to help teams better understand 
how data can inform classroom practice and 
improve student achievement. In Massachusetts 
state funds are available for professional develop- 
ment for pilot districts only. Once the initiative 
is launched statewide, districts will need to use 
local funds to implement this component. New 
Hampshire provides training in implementing a 
schoolwide culture that maintains a consistent 
focus on the use of data. New York’s regional 
school support centers provide professional 
development to teachers and administrators from 
low-performing schools in data-driven decision- 
making for content areas. In Vermont teachers in 
schools in need of improvement receive training 
in developing and maintaining professional learn- 
ing communities. 



Among the study 
jurisdictions only New 
Hampshire appears 
able to provide all four 
components for each 
state education agency 
initiative to every 
school in the state 



State implementation of key components. Among 
the study jurisdictions only New Hampshire ap- 
pears able to provide all four components for each 
state education agency initiative to every school in 
the state (see table 1). Massachusetts and Rhode 
Island are creating a longitudinal data system and 
data tools and preparing training on how to use 
them. Massachusetts is piloting 
a data warehouse and provid- 
ing professional development for 
data-driven decisionmaking in 
pilot districts. Once the initia- 
tive is implemented statewide, 
however, each district will have 
to fund its own training for using 
data analysis tools and profes- 
sional development in using data 



in decisionmaking. New York provides a data 
system, data tools, and professional development, 
although these are all part of separate initiatives 
and have little overlap. Maine is piloting a longi- 
tudinal data system and has online tools to access 
a limited warehouse of state assessment data, 
provides training in using those tools, and offers 
some help for schools in using data for decision- 
making. Vermont has a statewide data warehouse 
available to districts by subscription and also 
provides professional development to its low-per- 
forming schools in using data for decisionmaking. 
Connecticut is the only state focused solely on 
the process of data-based decisionmaking. It 
targets professional development in using data 
for decisionmaking to low-performing districts, 
although a centralized data system is planned for 
the near future. 

Although only New Hampshire is currently 
implementing a comprehensive data-driven 
decisionmaking system that is available to all 
schools, state education officials across the region 
noted the importance of having access to a data 
warehouse, being proficient in using data analy- 
sis tools, and understanding how to use data for 
decisionmaking. While initiatives in Connecticut, 
Massachusetts, and Rhode Island focus primarily 
on one component, state officials are aware of the 
need for a more balanced approach. 

The Massachusetts Department of Elementary 
and Secondary Education is funding a data 
warehouse and recognizes the need for profes- 
sional development on data use; however, districts 
are expected to invest in supports for teachers 
and administrators. As one department official 
explained, “districts have to come up with the 
training plan, professional development, and 
probably some consulting services” (interview, 
October 23, 2007). The University of Rhode Island 
has provided assistance to schools in using data 
for decisionmaking, with some support from 
the Rhode Island Department of Elementary 
and Secondary Education.® While Connecticut 
is currently focused on investing in professional 
development in using data for decisionmaking, it 
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has a federal longitudinal data system grant and 
is exploring ways to provide districts and schools 
greater access to data. An official from the Ver- 
mont Department of Education stressed the need 
for both technology and the capacity of educators 
to analyze data: 

/ really see there being two sides to this. . . . 

There is the tech side; every state gives a lot 
of data, and it’s overwhelming if you don’t 
know how to analyze it. Then there’s the team 
side; data is only as valuable as you have 
people to look at it. If you don’t have teachers 
[involved], then kid-specific solutions won’t 
be discussed. In fact, without this perspective 
[of teachers], people can make bad decisions. 
They might, for example, decide to change the 
curriculum, but it turns out it’s the kids who 
don’t eat breakfast who do badly. (Interview, 
October 23, 2007) 

Although Maine and Vermont are working toward 
implementation of all four components, limited 
funding has forced state education agencies to 
make choices about the components they provide. 
An official from the Maine Department of Edu- 
cation lamented the fact that their services are 
targeted to only a few schools: “Am I happy that we 
are hitting everybody and providing good instruc- 
tion in how to use data? No. We just simply don’t 
have the staff, but we do what we can” (interview, 
February 20, 2008). 

State education agency officials in the Northeast 
and Islands Region voiced the need for access to 
a data system/warehouse, for data tools, and for 
teachers and administrators to receive training in 
those tools, as well as for professional development 
in using data for decisionmaking. In every juris- 
diction but New Hampshire limited state funding 
and capacity appear to have hampered long-term 
goals of implementing a more comprehensive 
data-driven decisionmaking initiative. 

State education agency resources shape approaches 
to data-driven decisionmaking initiatives. Re- 
spondents noted competition for resources in 



In every jurisdiction 
but New Hampshire 
limited state funding and 
capacity appear to have 
hampered long-term 
goals of implementing 
a more comprehensive 
data-driven decision- 
making initiative 



implementing a com- 
prehensive data-driven 
decisionmaking ini- 
tiative. In particular, 
initiatives for a data 
warehouse and for tools 
to use the data warehouse 
are sometimes compet- 
ing priorities. This study 
reveals how funding and 
capacity shape state edu- 
cation agencies’ implementation of the data-driven 
decisionmaking initiatives provided to districts 
and schools. State education agency officials from 
several states reported that available funding and 
stair capacity have influenced decisions about 
which components to implement, who receives ac- 
cess to these supports, and which external service 
providers to work with to expand support. 



Funding. The source and amount of funding for 
state education agency initiatives can affect the 
services provided and who receives support. The 
Connecticut State Department of Education relies 
primarily on education funding from the state 
legislature and allocates its limited resources to 
support districts and schools with the highest 
needs. The Maine Department of Education offers 
training to a team of administrators from each 
district in using the online reporting tools for the 
state assessment, and staff are available to coach 
schools in using data for decisionmaking. Because 
of staff limitations, this coaching is available pri- 
marily to schools participating in specific grant 
programs (such as Title I and Reading First). 

Some districts in New York City have used Title 
II funding to support the development of school 
leaders (Darling-Hammond et al. 2007), which 
includes training in analyzing data and lead- 
ing school faculty in an evidence-based school 
improvement process. 



Massachusetts provides state funding to support 
its statewide data warehouse, which will be avail- 
able free of charge. It is piloting implementation 
with selected districts that have access to federal 
funds. Once the initiative is available statewide. 
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All state education 
agencies in the region 
have turned to external 
service providers 
to develop data 
warehouses and other 
technology solutions 
and to provide training 
in use of data tools 



the Massachusetts Department of 
Elementary and Secondary Educa- 
tion will provide guidance to 
districts on professional develop- 
ment in using the data tools and 
using data for decisionmaking, 
but districts will need to fund 
their own services. According to 
a Department of Elementary and 
Secondary Education official, the 
availability of federal funds has 
facilitated the pilot: “It’s been helpful for us to 
have partners ... in the districts who have actu- 
ally reached out and have applied for the Title II 
D grants” (interview, October 23, 2007). Vermont 
charges a subscription fee for access to its data 
warehouse and analysis tools, which are developed 
and maintained by a partner agency. Also for a fee, 
the partner agency provides professional develop- 
ment in using the data tools and analyzing the 
data. Vermont provides funds to low-performing 
schools for professional development on data- 
driven decisionmaking, but other schools and 
districts must pay their own way. With a federal 
longitudinal data grant. New Hampshire has been 
able to offer free statewide access to its data system 
and tools, training on tools, and professional de- 
velopment in data-driven decisionmaking. 



Capacity. State education agency support for data- 
driven decisionmaking in districts and schools is 
also shaped by the capacity of staff to work with 
districts and schools. The assessment division of 
the state education agency is commonly charged 
with helping local educators understand test items 
and how analysis of student scores can shape 
instructional practice. But these divisions are 
now being asked to do more with the same level 
of staffing. Although the state education agencies 
in the Northeast and Islands Region administered 
statewide assessments before 2001, under the 
NCLB Act they now test all students in grades 3-8 
every year and at least once during high school, 
dramatically increasing the number of assess- 
ments to be administered, scored, and analyzed. 
An official at one of the Maine Department of 
Education’s service providers. Measured Progress, 



previously assessment coordinator at the Maine 
Department of Education, noted the increased 
pressure on staff: 

[Departments of education staff] are being 
asked to do exponentially more with fewer 
resources. Prior to NCLB, most states tested 
at three grade levels. Overnight, they had to 
more than double the number of kids they 
were testing. In most cases, [departments of 
education] were doing it with the same and 
even reduced staff. And they were also trying 
to implement data warehouses. It’s insane 
what we ask state employees to do. (Inter- 
view, March 24, 2008) 

With inadequate staffing and the need for highly 
specialized skills in data-driven decisionmaking, 
state education agencies in the region are seeking 
vendors who can provide the needed services to 
districts and schools. All state education agencies 
in the region have turned to external service pro- 
viders to develop data warehouses and other tech- 
nology solutions and to provide training in use 
of data tools. The Massachusetts Department of 
Elementary and Secondary Education contracted 
with Cognos to develop a data warehouse, but 
three full-time staff members also work on it, and 
other staff are available for consultation. Vermont 
partnered with another organization to develop 
a data warehouse and focused state education 
agency resources and personnel on professional 
development for data-driven decisionmaking 
processes in low-performing schools. The external 
service provider is responsible for the data ware- 
house, analysis tools, and training on these tools. 
As one Vermont Department of Education official 
put it: “Being small, the department can’t do a lot 
of in-house services” (interview, October 23, 2007). 

State education agencies have also turned to 
external service providers to expand supports 
for implementing data-driven decisionmaking. 
Maine and the states that use the New England 
Common Assessment Program (New Hampshire, 
Rhode Island, and Vermont) have negotiated their 
large-scale assessment contracts with Measured 
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Progress to include some support for data-driven 
decisionmaking. The New York State Depart- 
ment of Education works with its regional school 
support centers to provide data-driven decision- 
making workshops to all of its low-performing 
schools. As a New York State Department of 
Education official stated: 

I oversee implementation of school improve- 
ment in New York City and there’s another 
office upstate in Albany that handles it, even 
though I do policy for everyone. We have 
regional support centers that we fund because 
we do not have the capacity to cover the state. 
(Interview, January 30, 2008) 

The Connecticut Department of Education, lack- 
ing adequate staff and expertise, partnered with 
the RESC Alliance to implement the professional 
development necessary to support its data-driven 
decisionmaking initiative for low-performing 
districts. 



WHAT SERVICE PROVIDERS DO STATE EDUCATION 
AGENCIES WORK WITH TO SUPPORT EDUCATORS' 
USE OF DATA-DRIVEN DECISIONMAKING? 

This section provides an overview of the external 
service providers mentioned in interviews with 
state education agency officials and then profiles 
three organizations that reflect the variety of ser- 
vices provided hy the full sample of organizations. 



ESP Solutions Group, and 
Measured Progress), four 
focus on creating data 
warehouses and tools to 
analyze data (Cognos, 

Pearson School Systems, 

Performance Pathways, 
and TetraData), and three 
focus on professional 
development for using 
data to improve school 
planning and classroom 
practice (BOCES, Con- 
necticut RESC Alliance, 
and Measured Progress). 

Appendix E provides ad- 
ditional information on the nine organizations. 



Three of the service 
providers identified by 
state education agency 
officials administer state 
assessments and provide 
support in interpreting 
test scores, four focus on 
creating data warehouses 
and tools to analyze 
data, and three focus on 
professional development 
for using data to improve 
school planning and 
classroom practice 



In-depth investigation of three service providers 

Three service providers were selected for profiling. 
Providers had to have worked with multiple juris- 
dictions in the region, have been in existence for 
two or more years, and have offered these services 
for two or more years. In addition, organizations 
had to offer professional development in using 
data to inform teaching and learning, not just sup- 
port on the other components. Connecticut RESC 
Alliance, Measured Progress, and Performance 
Pathways were selected on this basis (table 3). 
(Appendix B provides more detail on the selec- 
tion process; appendix E provides a more in-depth 
profile of each provider.) 



Interviews with state education agency officials in 
the Northeast and Islands Region identified nine 
organizations that were contracted to support 
data-driven decisionmaking initiatives at district 
and school levels (table 2). These organizations are 
not an exhaustive list of service providers in the 
region, but only those mentioned by interviewees. 
However, the organizations represent the range of 
services available in the region and the variety of 
organizations that exist to support state education 
agencies. Three of the service providers administer 
state assessments and provide support in inter- 
preting test scores (the Center for Assessment, 



Connecticut RESC Alliance. The Connecticut 
RESC Alliance, a regional education agency, is a 
consortium of all six regional educational ser- 
vice centers in the state. Since 2000, the regional 
educational service centers have met regularly to 
share problems and solutions and cost-effective 
methods for meeting the needs of public school 
districts through a variety of programs and ser- 
vices. Through the Connecticut Accountability for 
Learning Initiative, the RESC Alliance provides 
training for school data teams at low-performing 
districts, to help them build a data-driven 
decisionmaking process — the only data-driven 
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TABLE 2 

Overview of service providers identified by state education agencies, 2007/08 



State education 
agency 


External 

service provider 


Services provided 


Connecticut State 
Department of 
Education 


Connecticut Regional 
Educational Service 
Centers Alliance 
(ctrescalliance.org) 


• Programs and services tailored to the needs of particular schools and 
districts 

• Training in data-driven decisionmaking 

• Data warehousing 


Maine Department 
of Education 


Measured Progress 
{measuredprogress.org) 


• Development, implementation, and administration of K-12 education 
assessments 






• Professional development related to use of data tools, item analysis, 
and formative assessment 


Massachusetts 
Department of 
Elementary and 
Secondary Education 


Cognos 

(cognos.com) 


• Business intelligence software 


New Hampshire 
Department of 
Education 


Performance Pathways 
{perfpathways.com) 


• Data-analysis tools and training on tools 

• Professional development in creating a data-informed culture in 
schools 


New York State 

Education 

Department 


NY Board of Cooperative 
Educational Services 
(BOCES) 

{eboces.wnyric.org/wps/ 

portal/BOCESofNYS) 


• Technical assistance and technology supports to districts and schools 

• Professional development in data-driven decisionmaking 

• Professional development in data management 


Rhode Island 
Department of 
Elementary and 
Secondary Education 


ESP Solutions Group 
{espsolutionsgroup.com) 


• Data systems and psychometrics in K-12 education for federal, state, 
and local education agencies 

• Consulting and direct services 


TetraData 

{tetradata.com) 


• Data management tools, created expressly for the K-12 community 


Vermont Department 
of Education 


Center for Assessment 
{nciea.org) 


• Partner with states and agencies 

• Design and implementation of effective assessment and 
accountability policies and programs 


Virgin Islands 
Department of 
Education 


Pearson School Systems 
{pearsonschoolsystems. 
com) 


• Assessment development 

• Technology products to facilitate data storage and analysis 



Wofe.'The state education agencies each identified one primary provider of data-driven decisionmaking services, except for Rhode Island, which identified 
two providers. 

Source: Authors' compilation based on interviews with service provider and state education agency staff and material provided by service providers. 



decisionmaking component that the Connecticut 
State Department of Education implements. The 
RESC Alliance also provides technical assistance 
and training in district-level data warehousing 
and associated data analysis and offers a variety of 
customized work, such as spreadsheet and data- 
base training and custom report development for 
schools and districts. 



Although the RESC Alliance is an agency of the 
state of Connecticut and serves any district in Con 
necticut that requests support, it has also worked 
with districts in Massachusetts and is beginning 
to serve districts and schools in Rhode Island. The 
RESC Alliance is partially funded by the state but 
also raises revenue through outside grants and fee- 
for-service arrangements with districts. 
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TABLE 3 

Summary of three service providers, 2007/08 



Service provider 


Years in 
existence 


Type of service 


Clients 


Who pays? 


Connecticut RESC 
Alliance 


4 


• Training to develop 
school data teams to 
build a data-driven 
decisionmaking process 

• Training and technical 
assistance for district- 
level data warehousing 
and data analysis 
applications 


Districts: Any Connecticut 
district that requests 
services for data-driven 
decisionmaking 

States: Massachusetts and 
Rhode Island nonprofit 
education institutions 


• Partially funded by the 
state 

• Partially funded by 
revenues from delivery 
of services to districts 
and grant funds 


Measured Progress 


24 


• Assessment 
development 

• Professional 
development in 
analyzing assessment 
results, using data to 
improve instruction, 
and using formative 
assessment 


Districts: Professional 
development provider 
to Maine, Massachusetts, 
New Hampshire, and 
Vermont 

States: Assessment 
developer for Maine, 
Massachusetts, New 
Hampshire, New York, 


• State assessment 
contracts include 
some professional 
development to 
districts and schools 

• Districts and states 
contract directly with 
Measured Progress and 
pay for professional 



Rhode Island, and Vermont 
(in the region) and others 
outside the region (25 
states in all) (includes 
both assessment and 
professional development 
on using assessment data) 



development 



Performance 

Pathways 



4 (8 years 
for the two 
companies 
that merged 
to form 
Performance 
Pathways) 



Software for data 
analysis, assessment, 
and curriculum 
mapping and 
alignment to standards 

Professional 
development in using 
these software tools 
and creating a data- 
driven culture 



Districts: Connecticut, 
Massachusetts, New York, 
and Rhode Island (in the 
region) and others outside 
the region (47 states in all) 

States: State contract with 
New Hampshire 



Statewide contracts 
provide software with 
some professional 
development; 
additional professional 
development can be 
purchased from the 
company 

Districts with their own 
contracts pay for all 
their own services 



Source: Authors' compilation based on interviews with service provider and state education agency staff and material provided by service providers. 



Measured Progress. Measured Progress, a nonprofit 
company, provides both large-scale and alternative 
assessments, as well as professional development 
services in using data to improve student achieve- 
ment. Founded in 1983 as Advanced Systems in 
Measurement and Evaluation, Inc., the company’s 
stated mission is to improve teaching and learn- 
ing by providing assessment data customized to 
the needs of schools and by helping educators 
make the best use of the data to improve student 



achievement. Measured Progress has worked with 
more than 35 states and major districts to develop 
large-scale and alternative assessments. 

In the Northeast and Islands Region the company 
has created assessments for Maine, Massachu- 
setts, and New York and joint assessments (New 
England Common Assessment Program) for New 
Hampshire, Rhode Island, and Vermont. It also has 
contracted with districts in Maine, Massachusetts, 
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State education agencies 
rely on the specialized 
skills of these service 
providers to implement 
aspects of their data- 
driven decisionmaking 
initiatives at the district 
and school levels 



New Hampshire, and Vermont 
to provide professional develop- 
ment on using data and formative 
assessment to improve classroom 
practice. State assessment con- 
tracts with Measured Progress 
include some professional develop- 
ment to districts and schools on 
analysis of assessment data and its 
use in improving instruction. 



Performance Pathways. Performance Pathways, a 
for-profit technology company, focuses on bring- 
ing a data-driven informed culture to educators in 
schools. The company uses a suite of three prod- 
ucts to achieve this goal: Performance Tracker, 
Assessment Builder, and the TechPaths Cur- 
riculum Mapping System. Performance Tracker 
software allows users to access assessment data 
and student information in a central location and 
produce reports and graphs for analysis. Using As- 
sessment Builder, educators can create and score 
local benchmark assessments based on a library of 
items aligned to state standards. TechPaths helps 
educators create, map, and align curriculum and 
instruction to state standards. 



The company has been in existence for three years, 
the result of a merger between AlterNet Perfor- 
mance and TechPaths Company. The merger al- 
lowed the two companies to expand their offerings 
to include data tools, training on the use of tools, 
and support for creating a data-informed culture 
in districts and schools. Professional development 
provided by Performance Pathways focuses on the 



use of their data tools to support evidence-based 
decisionmaking and facilitate the development of 
curriculum and local assessments aligned with 
state standards. 

In the Northeast and Islands Region Performance 
Pathways has a state contract with New Hamp- 
shire and works with individual districts in Con- 
necticut, Massachusetts, New York, and Rhode 
Island. Statewide contracts provide districts with 
software and some professional development. 
Additional professional development can be pur- 
chased from the company. Districts also contract 
directly with Performance Pathways, using their 
own funds to pay for those services. 



Similarities and differences across 
the three service providers 

The three service providers offer different types 
of services to support one or more of the four 
components of a statewide data-driven decision- 
making initiative (table 4). While each of the 
three service providers has a different focus, state 
education agencies rely on the specialized skills 
of these service providers to implement aspects 
of their data-driven decisionmaking initiatives at 
the district and school levels. Two of the service 
providers offer services to support all four compo- 
nents, while one focuses on a single component. 
Two provide data tools and training in using the 
tools for analysis and reporting (Wayman et al. 
2005; Foley et al. 2008). All three provide at least 
some support to districts and schools in using data 
for decisionmaking or in creating a school culture 



TABLE 4 

Service providers' support for the four components of state education agency initiatives, 2007/08 



Service provider 


Centralized 
data system/ 
warehouse 


Tools for data analysis 
and reporting 


Training in data 
systems/warehouses 
and data tools 


Professional 
development in 
using data for 
decisionmaking 


Connecticut RESC 
Alliance 








/ 


Measured Progress 


/ 


✓ 


/ 


✓ 


Performance Pathways 


/ 


/ 


/ 


/ 



Source: Authors' compilation based on interviews with service provider and state education agency staff and material provided by service providers. 
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that uses data-driven decisionmaking (Boudett 
and Steele 2007; Knapp, Copland, and Swinnerton 
2007; Young 2006; City, Kagle, and Teoh 2005). 
And all three follow the “train the trainer” model 
as a way of leveraging districts’ professional devel- 
opment funds. 



The tools are housed on their web-based portal. 
Performance Pathways markets three data tools: 
one stores and analyzes data, another creates and 
scores local benchmark assessments, and a third 
facilitates mapping and aligning of curriculum 
and instruction to state standards. 



The three providers differ in their emphasis on 
developing a culture of using evidence for making 
education decisions and on promoting proficient 
and widespread use of data tools. The Connecticut 
RESC Alliance is more focused on building the 
capacity of data teams in schools, while Perfor- 
mance Pathways’ work is grounded in the devel- 
opment and use of data tools. Measured Progress 
falls between the two, with its assessments and 
analysis tools intended to build a climate of data 
use in schools. 

Centralized data system/warehouse. Two of the 
profiled external service providers provide a 
centralized data system/warehouse. Performance 
Pathways’ Performance Tracker software allows 
users to store multiple forms of data, so users can 
upload assessment data, student information, and 
other relevant data. The information is stored and 
linked together in a central location to support 
reporting. Measured Progress, as part of its work 
on assessments, helps clients create a web portal to 
facilitate customized data reporting. Data are gen- 
erated from the assessments created and scored 
by Measured Progress. The Connecticut RESC 
Alliance, which emphasizes the process of data- 
driven decisionmaking over the use of data tools, 
does not support services for creating a centralized 
data system. 

Tools for data analysis and reporting. The three 
service providers vary considerably in the use of 
technology to support data analysis and reporting. 
The Connecticut RESC Alliance offers services 
to support data-driven decisionmaking without 
the use of technology. District and school teams 
typically use paper copies of their state assessment 
reports and other data used in decisionmaking. 
Measured Progress is beginning to develop online 
data tools and generate accompanying reports. 



Training in data systems/warehouses and data 
tools. The two service providers that provide 
tools for data-driven decisionmaking. Measured 
Progress and Performance Pathways also provide 
training in their use. Both organizations stress 
the importance of this training for maximizing 
the effectiveness of the tools. Measured Prog- 
ress uses a train the trainer model for its online 
tools. Performance Pathways also offers a train 
the trainer model, providing training for school 
district representatives who then educate their 
colleagues on how to use the tools and data. This 
approach to professional development is increas- 
ingly common for districts (Dembosky et al. 2005) 
and indicates an orientation toward developing a 
process that will live beyond the initial workshops 
and participants. 



Professional development on using data for 
decisionmaking. All three service providers 
offer professional development on using data for 
decisionmaking. The Connecticut RESC Alliance 
trains teams of educators from low-performing 
districts to develop a data-driven culture through 
a collaborative process of examining data and 
student work. The training is intended to lead 
to changes in teacher instruction and student 
learning. In Connecticut a for-profit partner of the 
RESC Alliance and the Connecticut Accountability 
for Learning Initiative, the Leadership and Learn- 
ing Center, offers certification to participants to 
prepare them to lead data teams and teach their 
colleagues about using data for decisionmaking.® 



In conjunction with as- 
sessment work Measured 
Progress emphasizes 
changing the climate 
of schools and districts 
by using data to inform 



All three service 
providers profiled in the 
study offer professional 
development on using 
data for decisionmaking 
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instructional decisions. Measured Progress works 
with educators on how to interpret and use the 
assessment reports. Performance Pathways of- 
fers workshops to educators on how to use the 
information generated by their reports to make 
informed instructional decisions. 



CONSIDERATIONS FOR STATE EDUCATION 
AGENCY LEADERS DEVELOPING DATA- 
DRIVEN DECISIONMAKING INITIATIVES 

Several issues emerged from this study that may 
be relevant to state education agency leaders as 
they develop or expand initiatives to support data- 
driven decisionmaking. 



Adding components of a data-driven 
decisionmaking system 

Although resource constraints force choices about 
which components of a data-driven decisionmaking 
system to implement, state education agency leaders 
interested in supporting data use for decisionmaking 
in districts and schools might consider whether add- 
ing components could help to meet both intermedi- 
ate and long-term student achievement goals: 

• Where the focus has been on data warehousing, 
state education agencies might consider provid- 
ing more professional development on how to 
convert the stored data into usable knowledge 
that influences teaching and learning. 



depend on resources, including capacity and 
funding. 



Sources of funding 

Where inadequate resources impede implemen- 
tation of a comprehensive data-driven decision- 
making initiative, state education agencies could 
explore a range of funding options to augment 
their investments in data-driven decisionmaking. 
State education agencies might consider adopt- 
ing more creative ways to fund their data-driven 
decisionmaking initiatives, to enhance the sup- 
ports provided to districts and schools. Specifi- 
cally, state education agencies might take advan- 
tage of various federal grant programs (such as 
Title I, Title II, and Reading First) to augment state 
funding for data-driven decisionmaking. 

Strategies to enhance capacity 

The state education agencies in this report sup- 
port data-driven decisionmaking in districts 
and schools despite limited staff and sometimes 
limited expertise in developing tools or training 
educators. State education agencies have turned to 
external service providers to augment their data- 
driven decisionmaking initiatives. They might also 
consider negotiating additional related services 
under existing contracts to leverage current 
programs and offer greater support for data use in 
districts and schools. 



Where inadequate 
resources impede 
implementation of a 
comprehensive data- 
driven decisionmaking 
initiative, state 
education agencies 
could explore a range 
of funding options 
to augment their 
investments in data- 
driven decisionmaking 



• Where the focus has been on 
providing professional devel- 
opment to educators on using 
data for decisionmaking, state 
education agencies might 
consider investing in a data 
warehouse and data analysis 
tools to collect, organize, 
and analyze data for use in 
decisionmaking. 

Any decision to implement ad- 
ditional components will also 



CONSIDERATIONS FOR FURTHER RESEARCH 

This study describes how state education agen- 
cies in the Northeast and Islands Region support 
data-driven decisionmaking in districts and 
schools (see table 5 for a snapshot of services in the 
region). While it fills some gaps in the research, 
it raises additional questions about implementa- 
tion of data-driven decisionmaking. The research 
could be expanded by looking at state education 
agency support and implementation of data-driven 
decisionmaking in greater depth, at implementa- 
tion at different levels of the education system, and 
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at the impacts of data-driven decisionmaking on 
student achievement. 



Stages of implementation 

Many jurisdictions in the region are in the early 
stages of implementing these initiatives for 
data-driven decisionmaking. As state education 
agencies provide more support for data-driven 
decisionmaking, it would be interesting to revisit 
this topic: 

• How do states scale up their efforts to provide 
either comprehensive training or expand their 
support to all districts? 

• How do they support a range of data-driven 
decisionmaking activities, such as formative 
assessment? 



Different levels of the education system 

This study did not examine implementation of 
data-driven decisionmaking at the state, district, 
school, or classroom level. Exploration of imple- 
mentation at these levels of the education system 
could help to answer important questions. 

State education agencies. Although this study 
examined how state education agencies sup- 
port data-driven decisionmaking by districts 
and schools, it did not address how state educa- 
tion agency leaders might employ data-driven 
decisionmaking to inform their own decisions. 
The emphasis in the literature has been on district 
and school implementation rather than on state 
implementation. Some questions to consider: 

• What kinds of data-driven decisionmaking 
practices do state education agency leaders 
engage in? What kinds of data do they collect 
and use to make decisions on education poli- 
cies and practices? 

• What kinds of decisions are made by state 
education agency leaders who engage in data- 
driven decisionmaking practices? 



Districts. Although most data-driven decision- 
making research has focused on implementation 
at the school level, stakeholders in the region may 
wish to learn more about data-driven decision- 
making in districts throughout the Northeast and 
Islands Region. 

• How are districts supporting data-driven 
decisionmaking in schools? 

• Specifically, now that the Massachusetts 
Department of Elementary and Secondary 
Education has created its data warehouse in- 
frastructure, how are districts helping school- 
level staff use that data for decisionmaking? 

Schools. Avenues for further research at the school- 
level include: 

• Is data-driven decisionmaking increasingly 
embedded in school processes and classroom 
practice? 

• Are teachers and principals implementing 
the skills taught in professional development 
sessions? 

• What types of decisions are made by teach- 
ers and school administrators? Do they have 
access to the data needed to inform those 
decisions? 



Impact of data-driven decisionmaking 
on student achievement 

Once the data-driven decisionmaking initiatives 
are more fully implemented, there will be a need 
to investigate how data-driven decisionmaking 
affects student achievement. 

• Does data-driven decisionmaking help 
schools meet adequate yearly progress 
requirements? 

• What are the impacts of specific data-driven 
decisionmaking programs or practices on 
student achievement? 



Description of 

Jurisdiction data-driven decisionmaking initiative Targeted group Data coiiected Technoiogy avaiiabiiity Service provider 
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and supports targeted schools in using 
data to create school improvement 
pians 

Professional development to help 
special educators use data 
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Note: For further information on the materiai in thistabie, see appendix D. 

a. The service providers listed were identified by state education agency officials as the leading provider in the jurisdiction. 

Source; Authors' compilation based on interviews, documents provided by state education agency officials, and information available on state education agency web sites. 
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APPENDIX A 
GLOSSARY 

Culture of inquiry. A type of data-driven 
decisionmaking in which faculty create an organi- 
zational culture focused on using data and other 
evidence to shape instructional practices. 

Cycle of inquiry. An iterative process in which 
educators collaboratively analyze data to under- 
stand their instructional practice and develop a 
plan to support improvement efforts. 

Data. Information maintained by state educa- 
tion agencies, districts, schools, or teachers. Can 
include assessment data, school-level data on 
students and staff, demographic data, state test 
scores, and financial information. 

Data-driven decisionmaking. Systematic col- 
lection and analysis of various types of data — 
including input, process, outcome, and satisfaction 
data — by teachers, principals, and administrators 
to guide decisions to help students and schools 
succeed. 

Data-driven informed culture. A term used 
by Performance Pathways for data-driven 
decisionmaking. 

Data system. A centralized repository that 
combines data from multiple sources to provide 
an integrated view of data sources and a uni- 
form interface for data access (also called a data 
warehouse). 

Data teams. Small groups of teachers and admin- 
istrators working together on an ongoing basis to 
examine individual student work generated from 
common formative assessments. Data teams use 
collaborative, structured, scheduled meetings that 
focus on the effectiveness of teaching and learning. 

Data tools. Software that allows teachers and 
administrators to collect, organize, and analyze 
data for use in decisionmaking. Can be software 
on individual personal computers or online tools. 



User-friendly software allows users to easily ac- 
cess, manipulate, and analyze data to improve 
classroom practice. 

Formative assessment. All activities undertaken 
by teachers and their students to assess student 
understanding and to provide feedback for ongoing 
adjustments in teaching and learning. Timely ad- 
justments help students achieve targeted learning 
goals or standards within a set timeframe. 

Longitudinal data system. A system that holds 
and tracks multiple years of student and teacher 
demographic data, test scores and assessments, 
and other information. 

Low-performing school or school in need of 
improvement. A school that has not made ad- 
equate yearly progress for two consecutive years, 
as specified under the No Child Left Behind Act 
of 2001. Such schools are identified as “in need of 
improvement.” 

Multiple measures. The use of various assess- 
ments to characterize the performance of students, 
schools, and school districts. Measures for assess- 
ing student achievement can include student port- 
folios and exhibitions, performance assessments, 
teacher observations, and formal tests. School and 
district measures can include student growth mea- 
sures, promotion rates, attendance records, suspen- 
sion rates, graduation rates, enrollment in honor or 
advanced placement classes, and formal tests. 

Service providers. External organizations work- 
ing with states, districts, and schools to provide 
technology, professional development, and techni- 
cal assistance for data-driven decisionmaking. 

State data-driven decisionmaking initiatives. 

Initiatives conceptualized at the state level with 
a set of comprehensive goals, objectives, and 
purposes and with designated resources (such as 
funds and personnel) to support development and 
implementation. State education agency leaders 
must communicate such policies to districts in the 
state and offer them support. 
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Train the trainer model. A system in which some 
individuals are trained in a theory or technology 
and are then in turn responsible for training oth- 
ers in the same material. 



User conferences. Meetings that bring together 
educators who are working with a given product 
and allow them to learn more about the product and 
other products and how they work with each other. 
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APPENDIX B 
STUDY METHODS 

This project has two main purposes: to catalogue 
statewide data-driven decisionmaking initiatives 
being implemented in the Northeast and Islands 
Region through document reviews and interviews 
with key respondents from state education agen- 
cies and to catalogue service providers that sup- 
port data-driven decisionmaking initiatives in the 
region while profiling three such providers. 



Research questions 

Two research questions guided the study: 

• What state education agency initiatives support 
data-driven decisionmaking by districts and 
schools in the Northeast and Islands Region? 
What are their similarities and differences? 

• What service providers do the state education 
agencies work with to support educators in 
using data to inform education decisions, and 
what services do they provide? What are their 
similarities and differences? 



Sample selection 

The original sample included all nine jurisdictions 
in the Northeast and Islands Region. However, the 
research team was unable to schedule interviews 
with key respondents in the Puerto Rico Depart- 
ment of Education despite repeated attempts over 
several months to contact the department. All 
other jurisdictions in the region are profiled (see 
appendix D for state education agency profiles). 

The research team identified key respondents at state 
education agencies by drawing on existing relation- 
ships within the Regional Educational Labora- 
tory Northeast and Islands, through liaisons and 
researchers assigned to each state and through the 
team’s professional contacts from previous research 
in the region. In a few cases the team found respon- 
dents through web sites describing states’ data- 
driven decisionmaking initiatives. Depending on the 



state, the respondents held various roles in the state 
education agencies, including assessment coordina- 
tor, director of evaluation and research, information 
technology director, school improvement consultant, 
and information systems specialist (among others). 

Respondents at each state education agency 
provided names and contact information for 
lead service providers supporting its data-driven 
decisionmaking initiative with professional de- 
velopment for districts and schools. From this list 
a catalogue of nine service providers was created 
(see appendix E). This catalogue includes providers 
mentioned by state education agency respondents 
in interviews as the lead provider in their state and 
is not an exhaustive list of all providers working 
in the region. It includes one provider contracted 
with each state except Rhode Island, for which two 
were chosen because both had been selected by the 
state through the requests for proposals process. 

Three providers were then selected for more in- 
depth profiles. These providers met the following 
criteria: they worked with multiple jurisdictions in 
the region, they had existed for two or more years, 
and they had offered services for two or more 
years (that is, they were not starting up the activi- 
ties). This reduced the sample to eight providers — 
the New York Board of Cooperative Educational 
Services was excluded because it served only 
districts and schools in New York State. Organi- 
zations were selected from the remaining eight 
providers of professional development in using 
data to inform teaching and learning in addition 
to other services to districts in the region. That 
left four organizations — the Connecticut Regional 
Education Service Center Alliance, Measured 
Progress, Performance Pathways, and TetraData. 
TetraData and Performance Pathways offered 
similar services. As Performance Pathways offered 
a larger range of services, it was selected. 

The three service providers profiled in the report 
and in appendix F are: 

• The Connecticut Regional Educational Ser- 
vice Centers Alliance (provides professional 
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development on the process of data-driven 
decisionmaking). 

• Measured Progress (provides data warehous- 
ing services, data analysis tools, and profes- 
sional development on these tools and on 
data-driven decisionmaking). 

• Performance Pathways (provides tools for data 
analysis, reporting and curriculum align- 
ment, with professional development on these 
tools and on data-driven decisionmaking). 



Data sources 

To answer the research questions, researchers used 

data from four sources: 

• Publicly available information. This included 
reports from state education agencies and ser- 
vice providers on data-driven decisionmaking, 
descriptions of procedures and technical 
assistance to support data use, schedules of 
procedures, and information on state educa- 
tion agency and service provider web sites. 

• Supplemental documents. Respondents from 
both state education agencies and service 
providers offered additional documentation 
that was not publicly available. 

• Interviews. Semi-structured, open-ended 
interviews were conducted with key respon- 
dents from each state education agency and 
service provider. An application to Education 
Development Center’s Institutional Review 
Board for an expedited review of involvement 
of human subjects in a research project was 
approved, and researchers informed respon- 
dents of their rights and responsibilities 
through written informed consent forms. 

• Observations. When possible, interviews and 
reviews of official materials were supplemented 
with observations of professional develop- 
ment activities — a check on the self-reported 
information from the providers. Researchers 



observed two of the three providers profiled 
in the report. The timing of the project made 
it impossible for researchers to observe the 
workshops conducted by Measured Progress, 
which are held when state assessment data are 
released. Instead, researchers spoke with a par- 
ticipant from a school district that has worked 
with Measured Progress and an observer from 
the Maine state education agency to gather 
information about whether the workshop 
matched the service provider’s description. 

Two open-ended protocols were developed for the 
study, one for state education agencies and one 
for service providers, to guide data collection. 

The protocols for state education agency officials 
included questions addressing 1 1 key factors: 

• Motivation for the initiative. 

• Overarching goals of the initiative. 

• Data collected and used. 

• Tools used to collect, house, and disseminate 
data. 

• Users of data at different levels of the system. 

• Data analysis tools. 

• Decisions made using information gleaned 
from the data. 

• Leaders of the initiative. 

• Infrastructure in place to support data-driven 
decisionmaking. 

• Personnel supports in place. 

• Challenges to scaling up the initiative. 

The protocols for service provider personnel were 
designed to collect information on 12 key factors: 

• Objectives of services. 
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• Skills targeted. 

• Key components. 

• Users of the services. 

• Leaders of the services. 

• Amount of time that services had existed. 

• Numbers of schools and districts that pro- 
grams served. 

• Timeframe for the services. 

• Cost of the services. 

• Research base for the services. 

• Reasons states gave for selecting the services 
and service providers. 

• State leaders’ experiences working with the 
service providers. 

Some of these factors were based on the four di- 
mensions of data-driven decisionmaking that are 
designed to build and sustain capacity: leadership, 
personal supports, infrastructure, and program- 
matic content (Abbott 2008). 



Data collection methods 

Data collection began in September 2007 and 
ended in April 2008. Six sequential steps were 
taken to answer the research questions: 

• Collect publicly available information on 

data-driven decisionmaking initiatives. Public 
online documents, including state education 
agency web sites and web sites for state educa- 
tion agency partners (such as education agen- 
cies), were reviewed to gather information on 
jurisdictions’ data-driven decisionmaking 
practices. This helped to describe the initia- 
tives and, in some cases, to locate key contacts 
in each jurisdiction. 



• Conduct interviews with state education 
agency respondents. Three researchers 
investigated two jurisdictions each, and two 
researchers examined one each. Researchers 
held 45 -minute interviews with a key state 
education agency contact in each jurisdiction 
to gather descriptions of data-driven decision- 
making strategies. In all but one jurisdiction 
interviews were limited to one respondent. 

In Massachusetts two state education agency 
officials were interviewed. Most interviews 
were held by telephone, but interviews in 
Maine, Massachusetts, and Vermont were 
held in person at the state education agencies. 
In addition to answering questions about the 
jurisdictions’ data-driven decisionmaking 
initiatives, state education agency contacts 
supplied the names of data-driven decision- 
making professional development service pro- 
viders who worked with them and the names 
of their contacts at each service provider. All 
interviews were recorded and transcribed. 

• Gather publicly available information about 
service providers who work with state edu- 
cation agencies. Information from service 
provider web sites was used to create the 
catalogue of data-driven decisionmaking 
professional development service providers in 
appendix E. 

• Select three service providers for in-depth 
profiling. From the nine providers in the cata- 
logue, three were chosen for in-depth profil- 
ing using the criteria described above. 

• Interview key respondents at service providers. 
Telephone interviews were conducted with 
two key contacts at each of the three profiled 
service providers to gather information about 
their professional development services in 
data-driven decisionmaking. At Measured 
Progress, in addition, researchers interviewed 
a district administrator who had participated 
in the training and who demonstrated the 
data tools. Some phone interviews were fol- 
lowed up with in-person interviews to observe 
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service provider activities. All interviews were 
recorded and transcribed. 

• Observe professional development services. 
Teams of two researchers observed profes- 
sional development training programs or 
workshops. The observations allowed re- 
searchers to confirm information reported 
by the service providers. During the visits 
researchers asked contacts to provide ad- 
ditional documentation, including training 
and curriculum materials, course syllabi, and 
marketing information, thus gathering practi- 
cal information that could be useful to the 
state education agencies. For example, details 
on the cost, duration of services or activities, 
and size of typical workshops and trainings 
for a given provider can help states under- 
stand what to expect when working with such 
providers. Researchers also focused on learn- 
ing how service providers met the differing 
needs of jurisdictions. Interview transcripts 
and observation notes were systematically an- 
alyzed using the ATLAS-ti software program. 
Documents were used to further illustrate or 
explain issues raised in the observations and 
interviews. 

Because Measured Progress had already finished 
all its professional development activities at 
the time it was contacted, researchers reviewed 
workshop materials and spoke with two additional 
respondents who had attended its workshop in 
Maine. 

Researchers also interviewed two district officials 
in Massachusetts to understand how they access 
the state’s data warehouse initiative and how they 
use the available data for decisionmaking. 



Data analysis strategies 

The analysis of all collected data, including docu- 
ments and other provider materials, interview 
transcripts, and observational field notes, was 
based on the categories outlined in the original 
research proposal and in data collection protocols. 



The research team developed a code book or cod- 
ing scheme based on the factors that guided data 
collection and discussed and developed subcodes 
to allow a more nuanced analysis of the transcripts 
and notes. Most of the codes were developed 
before coding began, but a few subcodes were 
added during the discussions to ensure intercoder 
reliability. Subcategories were developed dur- 
ing analysis, though the overarching themes had 
been identified in the original proposal. The main 
coding themes, and examples of the subcodes that 
guided the analysis of transcripts and notes, ap- 
pear in tables B1 (state initiatives) and B2 (profes- 
sional development). Cross-state education agency 
analysis allowed the researchers to identify the 
four components of a data-driven decisionmaking 
initiative — a framework for the findings. 

To ensure uniform conclusions when coding 
transcripts and documents, just two coders were 
assigned to each of the two sets of interview 
transcripts (state education agency and service 
provider), but with three researchers participating 
in the coding. One researcher coded both sets of 
transcripts, partnering with a second researcher 
to code one set and with a third to code the other 
set. The two researchers coding each set worked 
closely to agree on interpretations of the qualita- 
tive data and to ensure consistency in coding. For 
each set, after coding a single transcript separately, 
researchers then discussed their individual coding 
of the transcript item by item. Each coder then 
coded a different subset of transcripts and notes. 

To develop the case profiles of statewide data- 
driven decisionmaking initiatives and triangulate 
evidence supporting the description of the initia- 
tives, transcript data were reviewed alongside both 
publicly available documents and those provided 
by state personnel. Information relevant to each 
code and category was synthesized thematically, 
based on the categories outlined in the analysis 
protocols (Abbott 2008), and the syntheses were 
used to describe each initiative. 

For the service provider case profiles, interview 
transcripts and observational field notes were 
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TABLE B1 

State initiative coding scheme 



1 Main theme 


Examples of subcodes 


Initiative 




Direction 






Requirement 






Focus 


Motivation for the Initiative 




Accountability 






Analyzing data 


Overarching goals of the Initiative 




Data storage 






Accountability 


Data collected and used 




State assessment 






Student information 


Tools used to collect, house, and 




Software from outside provider 


disseminate data 




Data warehouse 


Users of data at different levels of the system 




State department of education 






Principals 


Data analysis tools 




Use of software and reporting 






Use of professional development 


Decisions made using information gleaned 




Academic 


from the data 




Financial 


Leaders of the Initiative 




State department of education personnel 






District officials 


Infrastructure in place to support data- 




Professional development 


driven decisionmaking 




Data warehouse 


Personal supports in place 




Professional development centers 






State department of education personnel 


Challenges to scaling up the Initiative 




Money 






Culture 


Evaluation 




Initiative 






Outside provider 


Definition 




Data-driven decisionmaking 



Source; Authors' analysis. 



analyzed alongside documents provided by con- 
tacts at the service providers and coded for the key 
categories included in the interview and analysis 
protocols. Information for each code and cat- 
egory was synthesized thematically, based on the 
categories outlined in the analysis protocols, and 
compiled into each case profile. 

This systematic analysis of the collected data al- 
lowed the researchers to build case profiles of the 
state education agencies and the service providers. 



The team reviewed each profile, revised it based 
on comments from the team, and sent the revised 
profile to respondents for validation. Profiles were 
again revised after comments were received from 
respondents. 

After reviewing all the profiles, the team held sev- 
eral discussions to identify cross-profile similari- 
ties, differences, and challenges. These became the 
cross-profile themes that guided the discussion of 
findings in the body of the report, including the 
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TABLE B2 

Professional development services coding scheme 



Main theme 


Examples of subcodes 


States 




New York 

New England Common Assessment Program 


Overarching goals of the 
organization 




Provide technology-based products In data-driven decisionmaking to districts and 
states 

Provide professional development for using assessment data to districts and states 


Objectives of the services 




Create culture of data-driven decisionmaking In schools 
Make states better consumers of data 


Skills targeted 




Analysis of data 

Use of technology-based tools for data-driven decisionmaking 


Key components 




Training in process of data-driven decisionmaking 

Training in use of technology-based products for data-driven decisionmaking 


Users of the services 




District personnel 
Principals 


Leaders of the initiative 




State department of education personnel 
Outside service provider 


Amount of time services had been 




1-3 years 


in existence 




Varied for each service provided 


Numbers of schools and districts 




All districts in a state 


that programs served 




Some districts in a state 


Timeframe for the services 




Less than three months 
1-3 years 


Cost of the services 




Paid by state 
Paid by districts 


Research base for the services 




Based on work of data-driven decisionmaking experts 
Large-scale evaluation work 


Curriculum and content 




Analysis of data 

Use of technology-based tools for data-driven decisionmaking 


Trainers 




Company personnel 

District personnel previously trained by trainers 


Trainees 




District personnel 
Principals 


Kinds of data analyzed 




State assessment 
Student information 


Analysis tools 




Use of data teams 
Software-generated reports 


Decisions made using information 




Financial 


gleaned from the data 




Academic 


Medium of delivery 




In person, with technology 
Online 



(CONTINUED) 
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TABLE B2 (CONTINUED) 

Professional development services coding scheme 



Main theme 


Examples of subcodes 


Training process 


• Based on work of one data-driven decisionmaking expert 

• Based on work of various data-driven decisionmaking experts 


Materials provided to trainees 


• PowerPoint presentation from training 

• Activity sheets 


Follow-up training 


• Follow-up technical assistance as part of professional development 

• Follow-up technical assistance when requested 


Supports 


• Follow-up technical assistance 

• Technology-based tools 


Activities and goals 


• Data analysis and interpretation activity worksheets 

• Train other teachers and administrators in a district and school 


Connections with everyday 


• Use of real data 


practice 


• Access to real assessment items 


Implementation experience issues 


• Capacity 

• Teacher resistance 



Source; Authors' analysis. 



four components of a state education agency data- 
driven decisionmaking initiative. One member 
of the team took notes during the discussion and 
drafted the themes. Once the team had reviewed 
the draft, it met for another discussion to flesh 
out the themes and add documentation from the 
profiles. A similar process guided the development 
of the considerations for state education agency 
leaders and for further research. 



Study limitations 

The primary limitation of this project was the 
small number of respondents interviewed at 
each state education agency and service provider. 
Although the research team triangulated the 



information from the interviews with documenta- 
tion and observations, it is unclear whether the 
opinions of the respondents represented their col- 
leagues’ perspectives. 

In addition, the project did not explore the role of 
the districts or schools in supporting the imple- 
mentation of data-driven decisionmaking, nor 
did it consider the impact of data-driven decision- 
making initiatives on school improvement or stu- 
dent achievement. Finally, this project did not as- 
sess the effectiveness of any of the state education 
agency initiatives for data-driven decisionmaking 
or the impact of the professional development that 
was provided to build educators’ capacity to use 
data-driven decisionmaking. 
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APPENDIX C 

RESEARCH ON THE IMPLEMENTATION OF 
DATA-DRIVEN DECISIONMAKING, BARRIERS TO 
IMPLEMENTATION, AND SUPPORTS AVAILABLE 

This appendix examines the literature on data- 
driven decisionmaking. In particular, it explores 
what research says about the implementation of 
data-driven decisionmaking, the barriers to its 
implementation, and the available support to 
states, districts, and schools in implementing a 
data-driven culture. 



Implementation of data-driven decisionmaking 
at the district and school levels 

Studies of how data-driven decisionmaking is 
implemented show that districts and schools 
encourage various activities to build educators’ 
capacity to use data and evidence to improve stu- 
dent outcomes. The research suggests that, to ease 
the use of data-driven decisionmaking, districts 
and schools provide technology infrastructure and 
professional development to help educators collect, 
organize, and analyze data and to foster an envi- 
ronment that supports a culture of inquiry. Al- 
though most of these findings were gleaned from 
case studies of district and school implementation, 
one national survey of teachers has documented 
their data-driven decisionmaking practices. 

Education professionals collect various data, 
including test scores, classroom assessments, and 
information on the background characteristics of 
faculty, students, and schools (Gallagher, Means, 
and Padilla 2008; Wayman 2005). These data may 
be housed in data systems/warehouses (Marsh, 
Pane, and Hamilton 2006; Mills 2008; Wayman, 
Stringfield, and Yakimowski 2004), which facili- 
tate teacher and administrator access to multiple 
sources of data (Dembosky et al. 2005; Kerr et al. 
2006; Lachat and Smith 2005; Marsh, Pane, and 
Hamilton 2006; Storandt, Nicholls, and Yusaitis 
2007; Supovitz and Klein 2003; Wayman 2005). 

In 2006/07 nearly all school districts in a nation- 
ally representative sample surveyed by the U.S. 
Department of Education’s National Educational 



Technology Trends Study maintained at least some 
student data electronically and the majority (91 
percent) of teachers surveyed reported access to 
at least limited data on their students (Gallagher, 
Means, and Padilla 2008). 

In addition, many educators have access to data 
tools that can facilitate data-driven decision- 
making. These can include software programs and 
online data portals that provide more advanced 
and customized ways of analyzing and reporting 
data (see case studies by Larocque 2007; Storandt, 
Nicholls, and Yusaitis 2007; Chen, Heritage, and 
Lee 2005; Lachat and Smith 2005; Light et al. 2005; 
Murnane, Sharkey, and Boudett 2005; Mason 
2002). These software programs can either be sep- 
arate from data warehouses or used in conjunction 
with data warehouses or other types of databases. 
Training on both the data systems/warehouses 
and the analysis tools is commonly provided by 
the organization that developed the system or 
through a district’s central office staff (Foley et 
al. 2008). Training educators how to access and 
analyze data is an important part of developing 
data-driven decisionmaking among school or 
district staff— research shows that educators often 
lack an understanding of data, or the skills needed 
to track and analyze useful data (Earl, Katz, and 
Fullan 2006; Dembosky et al. 2005). 

Research on implementation of data-driven 
decisionmaking reveals that teachers and admin- 
istrators also receive professional development on 
data-driven decisionmaking. Schools with com- 
prehensive data-driven decisionmaking develop a 
culture of inquiry in which educators collabora- 
tively analyze data, to understand their instruc- 
tion practice and to develop plans to support 
school improvement efforts (Boudett and Steele 
2007; Knapp, Copland, and Swinnerton 2007; 
Young 2006; City, Kagle, and Teoh 2005; Lachat 
and Smith 2005; Murnane, Sharkey, and Boudett 
2005; Copland 2003; Feldman and Tung 2001). 

This cycle of inquiry helps schools evolve into 
learning organizations or “organizations where . . . 
people are continually learning to see the whole 
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together” (Senge 1990, p. 3). In these schools 
educators use data to clarify instructional prob- 
lems (Firestone and Gonzalez 2007). Professional 
development to support these processes follows 
the model of professional learning communities, 
which focus on the development of collaborative, 
reflective practices (Feger et al. 2008). In profes- 
sional learning communities for data-driven 
decisionmaking, often called data teams, coaches 
guide teams of educators through data-driven 
decisionmaking to help them use data to make 
and evaluate changes in instruction (Love et al. 
2008). 



Barriers to data-driven decisionmaking 

Research on data-driven decisionmaking in 
districts and schools also documents barriers to 
using data to improve student outcomes. A pri- 
mary barrier is that the available data may not be 
relevant to school and district improvement plans. 
For example, data used to demonstrate adequate 
yearly progress may not be appropriate for analyz- 
ing classroom practice (Mandinach 2008). Where 
teachers have access to student data, the data 
system might not provide tools to organize and 
analyze the data in a meaningful or relevant way 
(Gallagher, Means, and Padilla 2008). Many edu- 
cators face difficulties in transforming data into 
actionable knowledge (Massed 2001; Chen et al. 
2000; Gallagher, Means, and Padilla 2008; Hassler, 
Buck, and Torgesen 2004; Love 2004; Lang et al. 
2007), in part due to a lack of essential knowledge 
and skills to analyze data (Lang et al. 2007; Marsh, 
Pane, and Flamilton 2006). 



External supports for data-driven 
decisionmaking in districts and schools 

To overcome the barriers described above and 
to effectively use data-driven decisionmaking in 
districts and schools to improve student outcomes, 
educators not only must have access to data but 
also must know how to organize and analyze the 
data available. Often they need external sup- 
ports to develop their capacity to use data in their 
districts and schools, so a variety of external 



organizations provide professional development 
on data-driven decisionmaking to district and 
school administrators and teachers (Feldman and 
Tung 2001; Lachat and Smith 2005; Ikemoto and 
Marsh 2007). Such external support often comes 
from organizations that create data tools (Dem- 
bosky et al. 2005; Storandt, Nicholls, and Yusaitis 
2007; Wayman, Stringfield, and Yakimowski 2004) 
or from professional development organizations or 
universities that help districts and schools develop 
data-driven decisionmaking (Boudett, City, and 
Murnane 2005; Darling-Hammond et al. 2007; 
Thorn, Meyer, and Gamoran 2007). 

Although most research on external supports for 
data-driven decisionmaking focuses on profes- 
sional development organizations and universities, 
state education agencies have also recently begun 
to support data-driven decisionmaking in districts 
and schools (Dembosky et al. 2005; Massel 2001; 
Moss and Piety 2007; Reichardt 2000). Because 
these initiatives are new, there is little research in 
this area. A handful of descriptive studies were 
found that looked at state education agency initia- 
tives to support data-driven decisionmaking at the 
local level. They show that states have supported 
the needs of local educators in three primary ways: 
creating a policy structure to support and encour- 
age data-driven decisionmaking, providing data 
storage and analysis tools, and building the capac- 
ity of local educators to use data. 

One descriptive study, released prior to the imple- 
mentation of the No Child Left Behind (NCLB) 

Act, documents that the state education agency in 
Wyoming supported data-driven decisionmaking 
in three ways: by providing access to state assess- 
ment data (through a standards-based account- 
ability system), by requiring school improvement 
plans to include using data to measure progress, 
and by providing each district with Statisti- 
cal Package for the Social Sciences software 
and free training on that software (Reichardt 
2000). Another study, involving 100 interviews 
with educators in six districts and surveys of 26 
superintendents, detailed how the state education 
agency in Pennsylvania makes resources available 
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to all districts to support data-driven decision- 
making (Hergert et al. 2009). These supports 
include customized reports of state assessment 
data, an annual training seminar on data-driven 
decisionmaking, and help for districts to imple- 
ment the Pennsylvania Value Added Assessment 
System for growth modeling of individual student 
achievement. The Pennsylvania state education 
agency also focuses other resources — for example, 
supports for school improvement planning — on 
schools in need of improvement. Interviews with 
state education agency officials and a review of 
documents revealed that state education agencies 
in the Northeast and Islands Region support edu- 
cators in low-performing schools to develop their 



capacity to analyze and use data in their school 
improvement plans (Hergert et al. 2009). 

Data-driven decisionmaking at the local level is not 
new. But the need for the state education agency 
to play a role in supporting data-driven decision- 
making in districts and schools has increased 
dramatically in the past decade, with new state 
and federal external accountability systems. The 
increased nationwide emphasis on data-driven 
decisionmaking as a school improvement strat- 
egy, combined with a lack of information about 
state education agency supports for data-driven 
decisionmaking, makes this report an important 
contribution to data-driven decisionmaking efforts. 
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APPENDIX D 

PROFILES OF STATE EDUCATION AGENCIES' 
DATA-DRIVEN DECISIONMAKING INITIATIVES 

This appendix provides profiles of the state data- 
driven decisionmaking initiatives in Connecticut, 
Maine, Massachusetts, New Hampshire, New York, 
Rhode Island, Vermont, and the Virgin Islands. 



Connecticut 

The Connecticut State Department of Education’s 
statewide data-driven decisionmaking initia- 
tive targets low-performing districts. Under the 
department’s Connecticut Accountability for 
Learning Initiative (CALI), regional educational 
service centers train teams from schools in low- 
performing districts to analyze state assessment 
results and other data to improve instruction and 
student performance. The basic training for teach- 
ers and school administrators focuses on how to 
build and sustain data-driven decisionmaking in 
district- and school-level data teams. The Leader- 
ship and Learning Center, a partner, provides 
certification training for educators who plan to 
create a schoolwide culture of data use through 
professional development at their schools. 

State context. Connecticut has 1,111 schools serv- 
ing 575,058 students. The majority of students 
are White (67.0 percent), and the rest are either 
Hispanic (15.4 percent) or Black (13.7 percent). 
Roughly a quarter (26.5 percent) of students in the 
state are eligible for free or reduced-price lunch, 
and 44 percent of the schools are Title I. Con- 
necticut schools have long been high performing, 
obtaining scores above the country average on 
mathematics and reading for all administrations 
of the National Assessment of Educational Prog- 
ress (NAEP) since 1992. 

The Connecticut Mastery Test (CMT), a state assess- 
ment, is administered annually to all public school 
students in grades 3-8 to assess mathematics, read- 
ing, and writing skills. The Connecticut Academic 
Performance Test (CAPT), given to students in 
grade 10, is a natural extension of the CMT. The 



CAPT assesses and reports on student performance 
in four areas: mathematics, reading across the disci- 
plines, writing across the disciplines, and science. 

The state has been making major changes to the 
organization of the Connecticut State Department 
of Education and its relationship to school districts 
and schools, with a new commissioner appointed 
in May 2007 and new school reform legislation 
passed in June 2007. The Bureau of School and 
District Improvement and the Bureau of Account- 
ability, Compliance, and Monitoring were tasked 
with delivering “systematic and systemic” support 
to schools and districts. These bureaus work with 
all districts to provide data on student and district 
outcomes, processes to analyze data, methods of 
developing improvement activities, and training 
through the CALI. 

Overview of the initiative. The Connecticut State 
Department of Education’s statewide data-driven 
decisionmaking initiative is part of the CALI, 
which aims to advance learning for all students, 
particularly students from the 12 districts under 
corrective action. The Connecticut State Depart- 
ment of Education partnered with the regional 
educational service centers and the Leadership 
and Learning Center to provide support to dis- 
tricts and schools in six key areas (Connecticut 
State Department of Education 2008):^ 

• Data-driven decisionmaking data teams: using 
district and school data for analyzing, setting 
goals, and implementing research-based 
instruction strategies. 

• Making standards work: aligning school 
and district assessment and instruction and 
developing classroom-based assessments to 
monitor student progress. 

• Effective teaching strategies: examining ways 
to write lesson plans and deliver instruction 
using Dr. Robert Marzano’s research-based 
strategies (Marzano, Pickering, and Pollock 
2001, cited in Connecticut Department of 
Education 2008). 
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• Common formative assessments: collabora- 
tively designing assessments to improve learn- 
ing, as opposed to assessments of learning, 
and analyzing the results. 

• School climate improvement to support 
academic achievement: exploring practical 
structures and strategies that address bullying 
and school violence by creating safe learning 
environments. 

• Accountability in district and school improve- 
ment planning: creating a framework for a 
new accountability system. 

The CALI’s overarching goal is to reduce student 
achievement gaps. Its data-driven decision- 
making component, which began in 2004, 
provides districts with professional develop- 
ment and technical assistance in using data to 
improve pedagogy and learning. The profes- 
sional development and technical assistance are 
offered to Title I districts and schools identified 
by adequate yearly progress measures as in need 
of improvement. Ineligible districts and schools 
can participate for a fee, although in 2008 the 
professional development and technical as- 
sistance will be offered to the entire state. For 
districts that choose to participate, four days per 
year are set aside for professional development, 
and follow-up technical assistance is offered 
throughout the year. 

Three external service providers offer data-driven 
decisionmaking services to high-need districts in 
the state: the CALI, the Leadership and Learning 
Center (an educational professional development, 
publishing, and consulting organization based in 
Denver, Colorado), and the Regional Educational 
Service Centers (RESC) Alliance (see appendix E) 
with the State Education Resource Center (SERC). 
The CALI oversees implementation of the pro- 
fessional development and technical assistance, 
which are funded by the districts. The Leadership 
and Learning Center creates materials for the pro- 
fessional development training. The RESC Alliance 
and SERC deliver the training. 



The CALI offers two kinds of data-driven 
decisionmaking training: basic training and cer- 
tification training. The basic training is a two-day 
seminar for teachers and school administrators, 
including curriculum developers, superinten- 
dents, and principals. Participants learn first how 
to examine their own real student data using 
a six-step data-driven decisionmaking process 
(treasure hunting, analyzing needs, prioritizing 
needs, setting goals, identifying instructional 
strategies, and determining results indicators) 
and then how to develop and sustain this pro- 
cess in district- and school-level teams of fellow 
teachers and administrators. These data teams 
collaboratively analyze data and identify student 
strengths and weaknesses. Team members then 
come up with instruction strategies to address 
these as well as required learning standards.® 
Team members implement their strategies, moni- 
tor them, and discuss them at the next meeting, 
which should take place in their districts. 

The three-day certification training is for educators 
who plan to create a schoolwide culture of data use 
through professional development at their schools. 
During the training they learn more about the 
data-driven decisionmaking process taught in the 
basic training and about how to become effec- 
tive leaders of professional development in their 
schools. Participants bring their own classroom- or 
school-level data to be analyzed. Some attendees 
bring state assessment data. Others bring data on 
behavioral issues, such as tracking the number of 
classroom referrals and office referrals for students. 
In addition, some bring school climate survey data. 
The data are typically on paper or in a spreadsheet. 
No technological analysis tools are used. Data anal- 
ysis occurs through the six-step process mentioned 
above: users examine their data (treasure hunting), 
identify and analyze student needs, prioritize the 
needs, set goals for meeting them, identify instruc- 
tion strategies to target the goals they set, and de- 
vise ways to measure whether the goals have been 
met once the strategies have been tested (deter- 
mining results indicators). Actions resulting from 
this process include curriculum revision, program 
redesign, and funding redistribution. 
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Districts that participate in the professional de- Maine 

velopment receive two kinds of support: technical 
assistance and materials to facilitate data team 
work. Technical assistance, provided to districts 
throughout the year as a follow-up to the profes- 
sional development sessions, is based on the specific 
needs of each district that requests it. Once the 
training is complete, the materials provided during 
the training support the formation of data teams in 
each participant’s district. These materials include a 
booklet with an overview of the six-step data-driven 
decisionmaking process, activity worksheets for 
each step, a booklet describing the data teams and 
how they should work, and activity sheets to guide 
participants in implementing the teams in their 
districts. In addition, a DVD is provided with video 
clips of a sample data team. The initiative does not 
provide a technological infrastructure to house data. 

Next steps. Starting in fall 2008, the CALI planned 
to expand its services to all districts in the 
state, rather than providing them to only low- 
performing districts. 

Additional resources. These include: 

• The CALI web site at www.sde.ct.gov/sde/ 
cwp/view.asp?a=26I8&Q=32I754, techni- 
cal assistance standards at www.sde.ct.gov/ 
sde/cwp/view.asp?a=2618&q=322294, and 
data-driven decisionmaking and related 
training at www.sde.ct.gov/sde/cwp/view. 
asp?a=26188i:q=321744 and www.sde.ct.gov/ 
sde/lib/sde/word_docs/curriculum/cali/cali_ 
training_description.doc. 

• The Leadership and Learning Center web site 
at www.leadandlearn.com. 

• The Connecticut RESC Alliance web site at 
http://ctrescalliance.org/index.html. 

• The SERC web site at www.ctserc.org. 

• The web site for public summary performance 
reports for Connecticut’s statewide testing 
programs at www.CTreports.com. 



Both leadership and staff at the Maine Depart- 
ment of Education are aware of the importance of 
data-driven decisionmaking. Before the No Child 
Left Behind (NCLB) Act of 2001 took effect, Maine 
was developing a balanced accountability system 
that combined local assessments with statewide 
testing in various grades. Maine implemented the 
Maine Educational Assessment (MEA) in 1984 
and has since supported teachers in analyzing 
MEA test scores to improve classroom practice. As 
the department moves to implement annual NCLB 
testing requirements, its focus is shifting from 
performance assessments to longitudinal analysis 
of MEA data, with a federal grant supporting the 
development of a longitudinal data system. De- 
spite constrained resources and a small staff, the 
department encourages and supports data-driven 
decisionmaking within specific programs and 
provides some support to schools and districts for 
developing their capacity to analyze data. 

State context. Geographically, Maine is the largest 
state in New England, with a larger landmass than 
the rest of New England combined. That creates 
challenges to equitably serving students across 
the state. Maine’s population of 1.3 million is 96 
percent White, compared with 67 percent across 
the United States, and is largely rural. Although an 
increasing population of immigrants has settled 
in the state’s three urban centers, overall school 
enrollment is declining slowly. Public school 
enrollment declined from 213,867 in 1996/97 
to 193,335 in 2006/07 (Maine Education Policy 
Research Institute 2008). Historically Maine has 
had a high-achieving school system, measured in 
part by high rankings on the National Assessment 
of Educational Progress (Maine Education Policy 
Research Institute 2008). 

Overview of the initiative. Maine created the MEA 
in 1984 and, from its beginning, helped local educa- 
tors interpret their MEA scores and use the data to 
inform classroom practice. The state expanded this 
work in the 1990s. Maine’s education standards, the 
Maine Learning Results, were approved in 1997 and 
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included a local performance assessment designed 
to show what students knew and were able to do. 

In the late 1990s and early 2000s Maine helped dis- 
tricts develop local assessments and helped teach- 
ers and principals use the data to improve student 
achievement. With limited state resources, and 
given the difficulty of holding schools and districts 
accountable for myriad performance assessments, 
the Maine Department of Education is now focus- 
ing on creating a longitudinal data system that will 
support analysis of MEA scores at the student level. 

Although the department encourages data-driven 
decisionmaking in many of its state-sponsored 
programs and in its support of low-performing 
schools, it has no overarching, integrated policy or 
vision to encourage data-driven decisionmaking 
in districts and schools. Capacity is low: at the 
time of the study the state had several vacancies in 
data-related positions, no director of state assess- 
ment, and no National Assessment of Educational 
Progress coordinator, and both positions had been 
empty for some time. Also, Maine’s focus has been 
on supporting activities at the school level rather 
than on crafting a statewide data system. This 
focus is beginning to shift, with stronger external 
accountability measures imposed by the federal 
government and a federal grant supporting the 
development of Maine’s longitudinal data system. 

The MEA coordinator, one of a few department 
staff members dedicated to assessment-related 
work, is passionate about using data in the 
classroom — something she did actively when she 
was a teacher. “Having yearly data following kids 
through is powerful — if you know what you’re 
looking for” (interview, February 20, 2008). She 
regrets that, with limited staffing, state assessment 
staff cannot get out to schools: 

/ don’t work with individual districts that 
much. I think we have an obligation as a 
department to provide data-driven decision- 
making workshops, [to help them use] all the 
information that is available to them. "We get 
calls from people asking, “Can you help me? 

I have all this data, and I don’t know what to 



do with it; what I should be looking for?” We 
don’t have the capacity to do that on a large- 
scale basis. ... I help people on the phone all 
the time, but I can’t get out there. (Interview, 
February 20, 2008) 

Other department offices deploy content experts to 
work with teachers on item analyses. But, with- 
out an overarching state policy, the data-driven 
decisionmaking work can became sequestered in 
programs within the department. For example, 
the special education program supports districts 
in analyzing data on students with special needs, 
and the Title I office helps low-performing schools 
with data-driven decisionmaking. Staff from 
different program offices concur that MEA data 
are critical to their school improvement work and 
that data-driven decisionmaking is a key part of 
that work. Despite the lack of a formal, overarch- 
ing data-driven decisionmaking policy, informal 
communication and collaboration appear to 
support using data-driven decisionmaking across 
programs. 

The Maine Department of Education supports 
professional development for understanding and 
using MEA data, including both training and 
support in reading MEA reports and in using 
data to improve instruction. Support in reading 
MEA reports is provided by staff from the depart- 
ment assessment office and from the assessment 
company Measured Progress. The workshops, part 
of the state’s assessment contract, are provided 
free to districts. Other department offices support 
using the data to improve teaching and learning. 

Maine has contracted with Measured Progress 
to develop, administer, and score the MEA, to 
report test scores to each school in the state and 
to support district and school leaders in using 
the test data. Every fall, staff from Measured 
Progress and the Maine Department of Educa- 
tion’s assessment office lead report interpreta- 
tion workshops. These are held in five locations 
around the state to ensure that they are acces- 
sible to all Maine educators. In the fall of 2007 
Measured Progress rolled out a web-based portal 
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for accessing MEA scores, focusing professional 
development on helping school leaders both 
understand their scores and use the portal. The 
workshops are offered to district leaders, such 
as superintendents and curriculum directors. 
Some principals also attend. These leaders attend 
primarily to learn how to use data and under- 
stand adequate yearly progress in their schools. 
Such workshops have been offered since the state 
began administering the MEA in 1984. 

According to the MEA coordinator, with the new 
Measured Progress interactive web-based portal, 
a grade 5 teacher not only sees students’ MEA 
scores for this year but can check how they did 
on the grade 4 test. Yet a visit to a school district 
showed that the database accessible through the 
interactive portal is populated with only one 
year of MEA data. This is the first year Measured 
Progress has used the portal to provide assess- 
ment reports to schools, so only the current year 
of data is available online (interview, April 7, 
2008). Because of recent changes to state stan- 
dards and the MEA, there are no plans to add 
earlier test scores to the database. However, new 
test scores will be added in each subsequent year. 
Currently, the lack of longitudinal data limits the 
portal’s usefulness. 

Maine Department of Education staffers provide 
more in-depth professional development, focused 
on test content, to help faculty use MEA scores 
to improve instruction and to tailor it to each 
student. The department offers workshops for 
teachers in reading, writing, math, and science. 
Department staff realize that teachers will not use 
a lot of the MEA data, so they try to focus on item- 
level data and how these data can help teachers 
with instruction. 

Department staff go to schools, where they can tai- 
lor professional development to help school faculty 
learn from their classes’ responses to particular 
items. The staff explain how to look at specific 
released items and how to analyze data trends over 
time. They want teachers to be very familiar with 
the standards, or Maine Learning Results — the 



standards students are measured against on the 
MEA. In the words of the MEA coordinator: 

They are supposed to teach [to] standards, 
and the MEA is supposed to assess [by] 
those standards. So rather than teaching to 
the MEA, they should be teaching [to] the 
standards [that] the MEA will identify . . . but 
that’s a disconnect for people. They just want 
to get ready for the test. So we’ve been provid- 
ing release items for many, many years. . . . 

We encourage them to integrate as much as 
they can into the everyday class work [and] 
not just . . . get ready for the test a week before 
the test. (Interview, Eebruary 20, 2008) 

At the time of this study, though, there were only 
two content-area coaches to serve the entire state 
of Maine. 

Next step. The Maine Department of Education’s 
support for data-driven decisionmaking has 
evolved over time since the MEA was created in 
the 1980s. But such support appears to have grown 
stronger in recent years. The MEA coordinator 
attributed some of this to NCLB accountability 
requirements: 

We always encourage data-driven decision- 
making. That’s always our goal, internally as 
well as externally. I have seen a big change, 
anecdotally, since No Child Left Behind came 
in. The test finally means something to them. 
Before you’d get your bad score in the paper 
and you’d feel terrible, but now it’s changed. 

It’s huge. Now, they are actually looking at 
the data, and a lot of them didn’t before. 
(Interview, February 20, 2008) 

Despite this shift in attitudes, the coordinator 
recognized that the state could do more: 

Am I happy that we are hitting everybody 
and providing good instruction in how 
to use data? No. I’ve identified that as a 
need going forward, that we really need to 
collaborate. . . . We just simply don’t have 
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the staff, but we do what we can. (Interview, 
February 20, 2008) 

This issue has recently come to the fore in Maine. 

In June 2007 the Department of Education received 
a three-year, $3.2 million grant from the U.S. 
Department of Education to support the develop- 
ment of a longitudinal data system. The grant was 
intended to help states expand their existing data 
systems so they can better use data to meet report- 
ing requirements, support decisionmaking, and aid 
education research. States were selected in a com- 
petition and judged on their need for the project, 
its quality of design and management plan, and its 
promotion of timely generated accurate data for 
local, state, and federal reporting requirements. 

In February 2008 Maine contracted with Infinite 
Campus, an information management firm, to in- 
tegrate its current databases into one longitudinal 
data system. The Maine Department of Education’s 
leadership, along with its data management team, 
determined that the state needed a new data sys- 
tem to meet growing data needs. The new system 
will integrate MEA data with the student informa- 
tion system, Maine Educational Data Management 
Systems, and eventually will include staffing data. 
It is currently being piloted with a small number 
of schools. It will take several years to integrate all 
districts and schools into the system. 

Additional resources. These include: 

• Measured Progress at www.measuredprogress 
■org. 

• Infinite Campus at www.infinitecampus.com. 



Massachusetts 

Massachusetts is implementing the Educational 
Data Warehouse, a statewide initiative. The 
Massachusetts Department of Elementary and 
Secondary Education has purchased a license for 
data warehouse software that will house all the 
state-maintained data and allow school districts to 
load and analyze their own data. Staff at all levels 



(department, district, and school) will have access 
to the data for reporting and for making cur- 
ricular and pedagogical decisions. For three years 
Massachusetts piloted the initiative, and at the 
time of the study it was working toward statewide 
implementation, with 76 districts having access to 
the virtual warehouse. 

Participation in the initiative will not be 
mandatory — the state lacks funding to enforce it. 
The department obtained funding to purchase the 
online tool and pay a maintenance fee, to hire a 
few staff dedicated to the data warehouse, and to 
support school districts in the pilot phase. Once 
the initiative is statewide, districts that join will 
rely on their own budgets to build their capacity to 
understand, run, and use the software. 

State context. Massachusetts is the most populous 
of the six New England states, with more than 6 
million residents. The public education system 
serves nearly a million students in 388 school 
districts and 1,873 schools. In 1993 Massachusetts 
passed the Education Reform Act, which required 
the development of content and performance 
standards. As a result, the state developed the 
Massachusetts Comprehensive Assessment System 
(MCAS) to test all public school students on the 
state learning standards. Students are tested every 
year in reading and math in grades 3-8 and in 
high school. Science is tested in one elementary 
grade, one middle grade, and one high school 
grade. In addition to the content assessments, 
English language learner students are required to 
take the Massachusetts English Proficiency As- 
sessment (MEPA). 

Since 2001 the state has had a centralized system, 
the Student Information Management System 
(SIMS), to collect student background informa- 
tion. Each student has a unique identification 
number that allows the student’s SIMS and MCAS 
data to be tracked. At the time of the study, the 
state was in the first year of implementing the 
Education Personnel Information Management 
System, which gives each educator in the state an 
identification number. 
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Overview of the initiative. A few years ago, senior 
personnel at the Massachusetts Department of 
Elementary and Secondary Education concerned 
about increasing demands for using data in 
decisionmaking noted that the department col- 
lected and used data inconsistently and that data 
collected from the districts were not going back 
to the districts effectively In 2005 the department 
obtained internal funding to pilot the Educational 
Data Warehouse. The state invited districts to 
apply to the two-year pilot. At about the time when 
department officials were considering the data 
warehouse, district officials in the Northampton 
School District were searching for a way to better 
deal with the data they were collecting and with its 
various locations. In 2005 a Northampton admin- 
istrator met the state department official in charge 
of the Educational Data Warehouse initiative and 
shared the district’s vision for data collection and 
analysis. The district was awarded an Educational 
Data Warehouse grant. It and six other school dis- 
tricts became the first cohort to pilot the initiative. 

During the first six months of the pilot the 
districts and the department selected Cognos as 
the vendor for the data warehouse. Department 
officials met with representatives from Cognos to 
examine its data model and to discuss changes 
required to meet state and district needs. In 2006 
Massachusetts bought a statewide license for the 
warehouse. 

Once the warehouse was ready, the state depart- 
ment used it to make five years of its SIMS and 
MCAS data available to the pilot districts. Two 
districts with data management staff uploaded 
district-level data, such as students’ grades and 
courses. Many months were spent learning how to 
use the warehouse — running the reports, iden- 
tifying which local data could be uploaded, and 
learning how to upload them. 

Professional development has been integral to the 
project explains one distinct official: “Assuming 
that you can get your data loaded and you have 
access to data, what are you going to do with the 
data? . . . That is by far the most important and 



difficult question to answer and the most work” 
(interview, December 3, 2007). During the first 
two years of the grant administrative personnel 
from the districts received professional develop- 
ment from the Connecticut RESC Alliance (see 
appendix E) on using data for decisionmaking. 

The department got funding to expand the pilot 
for two additional two-year periods. May 2008 
marked the end of the initial phase of full rollout, 
with 76 districts having access to the warehouse 
and the department finalizing details of its 
expansion to more districts. Although use of the 
data warehouse and tools will not be mandatory, 
department officials expect that districts will take 
advantage of them. Though initiated and managed 
by the Massachusetts Department of Elementary 
and Secondary Education, officials from the seven 
school districts in the first pilot cohort — especially 
those from Amherst and Northampton — have 
been important in shaping it. Its main goals are 
twofold: to provide a platform to house data col- 
lected by the state, districts, and schools, making 
those data accessible in decisionmaking, and to 
provide software that allows state and district 
users to analyze their data. Different stakeholders 
will use the warehouse differently. For example, 
department staff will use it for reporting purposes, 
school personnel will look at data longitudinally, 
and teachers will access their students’ records 
from previous years to see their progress. 

At the time of the study, the warehouse contained 
state-maintained data, such as SIMS, MCAS, and 
MEPA data, and the state was uploading Educa- 
tion Personnel Information Management System 
data. The tool can incorporate other data, explains 
one department official: “We would love to get our 
finance data in there, [along with] . . . licensure 
data, academic program data, and professional 
development information. The types of things we 
could include and ultimately link are endless” 
(interview, October 23, 2007). Districts will also 
be able to load their own data, such as data on 
courses, grades, student attendance, or discipline. 
State department officials will notify districts 
when reviewing the districts’ uploaded data. 
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It is expected, once the data warehouse is imple- 
mented statewide, that people at different levels— 
policymakers, district officials, principals, school 
data teams, and teachers— will be able to access 
state data and their local data and to run reports 
tailored to their needs. 

The state education department bought a statewide 
license for the virtual data warehouse that will be 
available free to every school district in the state. 
Although the state is planning to provide direc- 
tions to districts about the warehouse, districts 
will have to budget to participate in the project 
and build district capacity around the warehouse. 

The warehouse has developed several personnel 
supports through pilot testing the initiative. Three 
fulltime state department staff personnel and two 
contractors work exclusively on the warehouse. 

In addition, weekly conference calls between 
department staff and districts in the first two 
pilot cohorts allow discussion of implementation 
issues. District and pilot department user groups 
gather feedback from end users and share best 
practices with them. Finally, a steering committee 
of department and district personnel meets to pri- 
oritize enhancement work for the data warehouse 
team. It grew out of a previous steering committee, 
drawn from the pilot districts, that met several 
times over two years to make decisions about 
implementation. 

Challenges to implementation. The warehouse has 
limited funding and will not be mandated once it 
is launched statewide. The pilots have shown that 
the initiative takes a good deal of districts’ time 
and resources to implement. One district official 
noted that, although the district received funding, 
it was a challenge to find “the time to clean the 
data, or the time to load the data, or the time to do 
professional development” (interview, December 3, 
2007). 

Once the initiative is launched statewide, the state 
department’s challenge will be to manage the 
expectations of the school districts. Districts must 
view the warehouse as a work in progress — for 



which cleaning and uploading data take time and 
resources — and learn how the uploaded data can 
be used for decisionmaking. Still, the state official 
in charge of the warehouse suggested that its ad- 
vantages outweigh the challenges: “I would rather 
see a model such as [a statewide Educational Data 
Warehouse] as opposed to a hundred different data 
warehouses being purchased by only the districts 
that can afford them” (interview, October 23, 

2007). 

Additional resources. These include the Massachu- 
setts Department of Elementary and Secondary 
Education, Educational Data Warehouse web page 
atwww.doe.mass.edu/infoservices/dw/. 



New Hampshire 

The New Hampshire Department of Education 
has made using data to improve instruction and 
student outcomes a priority, resulting in a compre- 
hensive approach to data-driven decisionmaking 
that includes tools for data storage and analysis, 
curriculum alignment, and the development of 
local assessments. The focus of its initiatives is on 
understanding individual student learning needs. 

State context. New Hampshire, a primarily rural 
state, has a large number of small schools and 
single-school districts. Districts are organized into 
school administrative units, each under a single 
superintendent. New Hampshire’s emphasis on 
local control has resulted in a recent education ac- 
countability law specifying that the state will not 
take over schools that fail to make adequate yearly 
progress. 

New Hampshire established its Statewide Educa- 
tion Improvement and Assessment program (RSA 
I93-C) in 1993, long before the NCLB Act, to set 
standards and require education improvement and 
assessment plans. Recent legislation (RSA 193 -H) 
sought to establish a single state accountability 
system for schools to be coordinated with the 
NCLB Act. This system seeks both to meet NCLB 
requirements and to go beyond them by including 
all public schools, both Title I and non-Title I. 
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New Hampshire has joined with Rhode Island and 
Vermont to develop the New England Common 
Assessment Program (NECAP), an annual assess- 
ment developed to comply with NCLB legislation. 
NECAP has been administered each fall in grades 
3-8 since 2005. In fall 2007 it was given in grade 
11. New Hampshire educators are also construct- 
ing course competencies at the secondary level 
that link content to skills and use multiple mea- 
sures to evaluate student proficiency. 

Since 2005 the state has emphasized educating the 
whole child. Its Follow the Child initiative fosters a 
personalized learning experience for each student 
and challenges districts to measure growth in and 
develop support systems for the personal, social, 
physical, and academic aspects of each student’s 
life. Participation by districts, schools, and indi- 
vidual educators is voluntary. 

Overview of the initiative. Data collection and use 
have been the focus of New Hampshire’s approach 
to school improvement for the past five years. Ac- 
cording to the leader of data use initiatives within 
the New Hampshire Department of Education, 
about three years ago the department recognized 
that its methods for data collection and data dis- 
semination to districts did not make good analysis 
likely at the district and school level. For example, 
districts were unable to compare year-to-year 
assessment results or to calculate graduation rates 
meaningfully. 

These deficiencies led New Hampshire to launch a 
coordinated statewide effort in 2005 to collect data 
that would allow for meaningful data analyses by 
districts and schools and to train users in data- 
driven decisionmaking. These efforts were aided 
by funding from the U.S. Department of Education 
for data collection and decisionmaking through a 
statewide longitudinal data system grant of $3.2 
million over three years. This grant is being used 
to improve data warehouse and analysis tools, to 
fund the Follow the Child initiative (explained 
below), and to collaboratively develop a database 
to connect school and postsecondary data on 
students and teachers. 



New Hampshire exceeds NCLB requirements 
through a joint initiative of the governor and 
the commissioner of education called Follow the 
Child, which brings together several data collec- 
tion and use programs to track each student’s 
progress toward (at least) proficiency on local 
and state assessments. Participation in Follow 
the Child is voluntary for districts (called school 
administrative units in New Hampshire), approxi- 
mately half of which were participating at the time 
of the study. 

New Hampshire is committed to the use of data 
to guide school improvement. A New Hampshire 
Department of Education publication states: 

All of us must use school performance data to 
continuously improve the education system 
in our state. Making this data available to 
the public is essential. Providing every New 
Hampshire child with a quality education is 
a result of assessing and understanding the 
situation, working together in our commu- 
nities, in our classrooms, and in our own 
homes to make a difference. (New Hampshire 
Department of Education 2006) 

The department’s vision for data-driven decision- 
making in the state is to enable districts to analyze 
assessment data and other data in various queries, 
to support district efforts to develop and analyze 
their own local assessments, and to link these data 
to curriculum and lesson planning tied to state 
standards. 

New Hampshire’s Initiative for School Empower- 
ment and Excellence system gives a unique iden- 
tification number to each student in New Hamp- 
shire, letting districts easily retrieve data for all 
their students. The U.S. Department of Education 
longitudinal grant will allow the state to expand 
the database to include postsecondary students. 

Statewide data collection efforts have been led by 
state education agency personnel. Professional 
development for district and classroom data use, 
led by New Hampshire’s five regional professional 
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development consortiums and centers, offer basic 
and advanced data analysis training for schools 
and districts. 

New Hampshire collects a wide range of data with 
its statewide data warehouse and analysis tool, 
Performance Tracker. All school administrative 
units use Performance Tracker. It has three mod- 
ules: Performance Tracker stores assessment and 
other data and facilitates analysis through various 
queries. Assessment Builder allows districts to 
develop customized assessments, and Content Li- 
brary facilitates linking assessments to curriculum 
and lesson plans. All the administrative units have 
free access to the first module. About 15 percent 
of the units (including Manchester and Nashua, 
the largest in the state) have paid for access to the 
other two, and many more units have expressed 
interest in accessing them. Funding such access 
through the federal longitudinal data grant has 
been discussed by the state department. 

State department staff reported that one strength 
of Performance Tracker is the range of data it can 
store and analyze. New Hampshire’s data-driven 
decisionmaking efforts have focused on making 
a wide range of data available for analyses. Types 
of data that districts can put into Performance 
Tracker include scores on NECAP, the Northwest 
Evaluation Association assessments (administered 
by about half of New Hampshire’s school admin- 
istrative units, these measure student instruction 
level and growth, to help teachers tailor instruc- 
tion to individual needs), the Dynamic Indicators 
of Basic Early Literacy Skills (DIBLS) test, and the 
Developmental Reading Assessment, along with 
data on attendance and academic, social, personal, 
and physical development from the follow the 
Child initiative. 

The New Hampshire Department of Education has 
two types of infrastructure to support data-driven 
decisionmaking: its data warehouse facilitating 
data storage, retrieval, and analysis— described 
above — and policies that promote data-driven 
decisionmaking in schools and districts. The 
policies focus on tying funding and adequate 



yearly progress reporting to data collection and 
analysis — to make schools and districts aware of 
the importance the state places on data use and 
to give them an incentive to analyze their data. 
Recognizing the extra effort and training commit- 
ments districts must make to learn data-driven 
decisionmaking, department staff emphasize that 
their role is to provide customer service to help 
districts progress toward effective data use. The de- 
partment answers immediate, individual questions 
from districts and schools on data collection and 
provides technical help with the data warehouse. 

In addition. New Hampshire’s five professional 
development consortiums and centers provide 
training in data-driven decisionmaking. The 
training is voluntary and funded by the districts. 
Thus, the consortiums and centers are motivated 
to cater training to district needs. 

Challenges to implementation. New Hampshire 
has a strong tradition of local control. This allows 
schools to adapt to local circumstances, but it also 
presents challenges to statewide education initia- 
tives. for data-driven decisionmaking it means 
that school administrative units use different 
assessments— NECAP is the statewide assess- 
ment developed in response to NCLB legislation, 
whereas other assessments such as the Northwest 
Evaluation Association’s are used voluntarily by 
local educators. Similarly, professional develop- 
ment is decentralized, differing by area. With the 
state lacking personnel to provide professional 
development on data use, much of the state’s 
formal training in data-driven decisionmaking 
is provided at the local level. Although the state 
department has provided supports for data-driven 
decisionmaking, in New Hampshire, it is difficult 
to determine how much data-driven decision- 
making is being implemented in school admin- 
istrative units and schools given the strong local 
control in New Hampshire. 

Additional resources. These include New Hamp- 
shire Department of Education at www.ed.state. 
nh.us/education/ and Performance Pathways at 
www.perfpathways.com. 
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New York 

New York has a strong history of using data-driven 
decisionmaking to inform education decisions at 
different levels of the system. Although the New 
York State Education Department has no formal 
statewide data-driven decisionmaking policy for 
districts and schools, it encourages such decision- 
making in its support of low-performing schools 
and through state-sponsored and other agency 
programs. Those programs include New York’s 
statewide data reporting system, nySTART, and 
the services provided by the Board of Coopera- 
tive Educational Services and the New York Data 
Analysis Technical Assistance Group. 

State context. New York contains some of the most 
densely populated areas in the Northeast and 
Islands Region, includes a wide range of urban, 
suburban, and rural schools serving nearly 2.8 
million students (U.S. Department of Education, 
National Center for Education Statistics 2007). 

The five largest districts— New York City, Buffalo, 
Rochester, Syracuse, and Yonkers — make up 42 
percent of the state’s public school population. 

New York has a very diverse population of stu- 
dents. A little over half (52.7 percent) are White, 
whereas the majority of the other half are either 
Hispanic (20.1 percent) or Black (19.8 percent). 
Most (68.2 percent) of the schools in the state are 
Title I, and almost half (44.8 percent) of the stu- 
dents are eligible for free or reduced-price lunch. 

Overview of the initiative. The Elementary, 

Middle, Secondary, and Continuing Education 
department of the New York State Department of 
Education established the Office of School Im- 
provement and Community Services (OSICS) in 
2003. The OSICS is charged with designing and 
implementing the state’s system of support for 
low-performing schools. Its overarching goal is to 
close the achievement gap in English language arts 
and mathematics for all students, including NCLB 
subgroups. 

In 1998 New York became one of the first states 
to have a system of accountability. With the 



accountability plan came the School Under Reg- 
istration Review process, which identifies schools 
performing furthest from state standards — based 
on state assessment scores — and requires that they 
create comprehensive improvement plans. The 
state education department has strong expecta- 
tions that districts and schools include data-driven 
strategies in their improvement plans. 

The seven regional school support centers are the 
primary means through which support is provided 
to New York’s high-priority districts and schools. 
Established in 2003 along with the OSICS and 
funded by the state education department, the 
centers provide technical assistance and support to 
schools and priority districts, and they coordinate 
the implementation of the Reading First initiative 
and the reauthorization of the Individuals with 
Disabilities Education Improvement Act (IDEA 
of 2004).® One center serves New York City, and 
the remaining six are spread throughout the state. 
The centers work with four other department- 
supported regional network partners to deliver 
services to schools and districts. 

Since the inception of the OSICS and the regional 
centers, low-performing schools have received 
substantial professional development focused on 
data-driven decisionmaking. Low-performing dis- 
tricts and schools, which must include data-driven 
strategies in their improvement plans, are encour- 
aged to participate in data-driven decisionmaking 
professional development. When developing the 
plans, the districts and schools are expected to 
use data from multiple assessments, which can 
include standardized assessments — with data that 
can be disaggregated by student subgroups — and 
benchmark assessments demonstrating student 
progress toward meeting learning standards. 

Other assessments must be used to measure the 
impact of curriculum and instruction delivery in 
core academic areas. 

These data can help schools revise their education 
priorities. Once priorities are established, they 
become the core of each school’s comprehensive 
education plan, including specific annual goals. 
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measurable objectives, and action plans for ob- 
servable and effective strategies to improve student 
achievement. 

One approach used to provide data-driven 
decisionmaking to low-performing schools in New 
York is through the reading, mathematics, science, 
English as a second language, and special leader- 
ship institutes. Held several times a year, these 
are offered by the OSICS with the New York City 
Department of Education, Fordham University 
Graduate School of Education, and the New York 
Comprehensive Center. Representatives from low- 
performing schools are required to attend the in- 
stitutes, at which the representatives attend work- 
shops led by researchers, professors, consultants, 
professional developers, and other educators on 
integrating research-based strategies and practices 
into teaching and learning. Many workshops teach 
participants to use data for decisionmaking in 
particular content areas. For example, a workshop 
at a December 2007 institute addressed the align- 
ment between assessments (formative and sum- 
mative alike) and learning standards for English 
language arts and the use of assessment informa- 
tion to drive instruction. Another session taught 
participants how to identify student weaknesses in 
mathematics by analyzing state assessment items 
and to use the information to derive instruction 
strategies targeting such weaknesses. 

Another approach to professional development in 
data-driven decisionmaking for low-performing 
schools is through the New York State Teacher 
Centers. The OSICS placed these on-site profes- 
sional development centers in low-performing 
schools, where they help teachers to use data. The 
state has paid for mathematics and literacy tutors 
for each school without a Teacher Center, to help 
teachers examine data more closely and use it to 
inform instruction. 

The professional development in data-driven 
decisionmaking provided to New York’s districts 
and schools over the past decade or so has contrib- 
uted to changes in the cultures of schools, shift- 
ing them toward knowing their data and being 



motivated to engage frequently with those data. 
Not only have teachers become better consum- 
ers of data but the state department’s associate 
commissioner finds that leaders in districts and 
schools are better informed about their data: 

When you go into our lowest performing 
schools . . . particularly in elementary schools, 
you will find that principals can talk to you 
about data. To me, this shows real change. . . . 
Before, you would have to find a classroom 
teacher to learn those things, if you were 
lucky. (Interview, January 30, 2008) 

Data-driven decisionmaking by district and school 
leaders is supported and encouraged in New York 
through avenues other than the initiative for 
improving low-performing schools. One data re- 
porting and analysis tool is available to all school 
leaders, and two types of networks or groups 
encourage and support districts and schools in 
their use. 

Through the statewide data reporting system, 
nySTART, the state education department encour- 
ages managing and reporting state assessment 
data. Formerly housed in 12 regional information 
centers, such data are now stored in the New York 
State Testing and Accountability Reporting Tool, 
or nySTART, a web site that gives school leaders 
(including district superintendents, principals, 
and teachers) access to reports on New York’s 
standardized test scores. The change reflected a 
desire to have a standardized online tool for the 
regional information centers and Boards of Coop- 
erative Educational Services (see below) to work 
with state assessment data. Run by the Grow Net- 
work and McGraw-Hill, nySTART is funded by the 
state department of education. The nySTART web 
site contains detailed reports on test results for 
assessments, including the New York State Testing 
Program, New York State Alternative Assessment, 
the New York State English as a Second Language 
Achievement Test, and the New York State Regents 
Examinations. District and school administrators, 
as well as teachers, are expected to use the data 
reports to inform education decisions. 
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The nySTART system provides verification, as- 
sessment, and accountability reports. Verifica- 
tion reports let school district officials review 
and verify the accuracy and completeness of 
data. Assessment reports provide summary- and 
individual-level performance in PDF for English 
language arts and mathematics assessments in 
grades 3-8 and for the New York State Alternative 
Assessment. The data can be analyzed in a variety 
of ways. For example, teachers and administrators 
can view and sort performance data by district, 
school, grade, and student. They can analyze ag- 
gregate performance and examine data by perfor- 
mance level, content strands and standards, and 
test item. Data can be summarized for subgroups 
(disability status, race/ethnicity, English profi- 
ciency, and so on). Last, the accountability reports 
provide information about school status under the 
state and federal accountability systems, including 
school profile information, school accountability 
status, and performance. 

New York’s regional school support centers 
work with various types of partners that are 
not department-funded to provide professional 
development in data-driven decisionmaking. One 
such type is the Board of Cooperative Educational 
Services (BOCES), regional networks that provide 
services including student supports, professional 
development, technical assistance, and technology 
supports that individual districts cannot provide. 
The state’s 37 boards are funded by the state and 
by districts, whose superintendents govern and 
determine services. BOCES provides data-driven 
decisionmaking services based on regional and 
district need. Eor example, in districts with the 
capacity to manage, interpret, and gather informa- 
tion from data to inform instruction, the boards 
may ease and enhance those tasks. Other districts 
lacking such capacity may receive professional 
development and technical assistance. FFoused in 
the boards are the 12 statewide regional informa- 
tion centers, which provide participating districts 
with technology services for instruction and 
administration as well as training and support for 
teachers, students, and administrators. The centers 
store and process all the data required for the state 



and provide customized professional development 
on data management and analysis to districts. 

The New York Data Analysis Technical Assistance 
Group, developed in 2000 at a Comprehensive Dis- 
trict Education Planning conference and funded 
by the state’s education department, grew out of a 
shared interest in creating data systems that would 
help teachers and other educators, rather than 
only those with technical expertise. The group is a 
research-based learning community, open to edu- 
cation professionals involved in supporting data 
use to improve instruction around the state. Mem- 
bers come from organizations including school 
districts, BOCES, special education training and 
resource centers, regional school support centers, 
regional information centers, the state education 
department, and the private sector. Members may 
be involved in a variety of data-driven activities, 
such as building data warehouses in the informa- 
tion centers or providing test scoring and data 
analysis to districts. The goal of the group is to 
“provide a leadership role in identifying, catalog- 
ing, and modeling best practices in the analysis 
and use of data for New York schools” (New York 
Data Analysis Technical Assistance Group 2008). 
The group holds meetings four times a year, with 
biannual conferences to discuss ways of building 
its capacity to provide data-driven decisionmaking 
services and supports. It also has an email list 
that is free to members, where members post 
questions and answers related to data-driven 
decisionmaking. 

Next steps. The New York State Education Depart- 
ment has long understood the need for consistent 
data-driven decisionmaking in districts and 
schools. It continues to build on its data-driven 
decisionmaking initiatives, for example, by plan- 
ning a new initiative to create an integrated data 
system for PreK through higher education. The 
new initiative was called for by the New York State 
Board of Regents in fall 2006. Its action plan calls 
for the department to create the new data system 
with the goal of using data to improve graduation 
rates in high school and in higher education (New 
York State Board of Regents 2006). 
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Additional resources. These include: 

• The New York State Education Depart- 
ment School Comprehensive Educa- 
tional Plan at http://schools.nyc.gov/NR/ 
rdonlyres/75097ED5-478D-4129-AE78 
-EF0442C60B3D/32663/RequiredCEP 
Appendicesfor20072008ReleasedMay22007. 
doc and the District Comprehensive Edu- 
cational Plan at http://schools.nyc.gov/NR/ 
rdonlyres/138A84DB-3E16-418F 
-AF7C-A90B266FFAFE/32645/ 
DCEPTemplate200708v807.doc. 

• The Board of Cooperative Educational Ser- 
vices web site at www.monroe2boces.org/ 
statewide.cfm. 

• The Data Analysis and Technical Assistance 
Group web site at www.datag.org. 

• nySTART web site at www.nystart.gov. 

• Introduction to nySTART (PowerPoint) 
at http://emsc32.nysed.gov/irts/nystart/ 
PD-0713-Intro-to-nySTART.ppt. 

• P-16 Education: A Plan for Action at http:// 
usny.nysed.gov/summit/p-16ed.pdf 



Rhode Island 

The Rhode Island Department of Elementary and 
Secondary Education is creating a central data 
warehouse. Housing all education-related data, 
including state assessments, student information, 
teacher information, and financial information 
related to schools, the warehouse will allow users 
to access data with up-to-date graphs and figures 
as well as user-designed reports. 

It will be available in different ways to users at all 
levels of the system: parents, teachers, administra- 
tors, education department employees, and the 
state legislature. The state is focusing professional 
development on helping users in all school dis- 
tricts understand the new data warehouse. 



State context. Rhode Island is the smallest in 
area of any state. It is highly urban and densely 
populated, with a public school student population 
of 152,422. Most students are White (70.5 percent), 
with Hispanics (17.3 percent) the largest minor- 
ity group. Close to half (40.9 percent) of the state’s 
schools are Title I, and just over a third (34.9 per- 
cent) of the student population is eligible for free 
or reduced-price lunch. 

In 2006 the state began to build a data ware- 
house for all education data. Formerly in several 
locations, data would now be available to users 
through one consolidated source (with seven com- 
ponents). Users can obtain up-to-date data in a 
user-friendly form and can generate reports based 
on the data. 

Overview of the initiative. Rhode Island’s work 
on a data warehouse began with district officials 
looking for a way to analyze data and use that data 
to inform decisions on education. As districts and 
the state education department began looking 
into systems, they realized that they were missing 
a foundation: a statewide student identification 
system. After such a system was created in March 
2004, a two-year project to develop the warehouse 
was funded through a statewide bond referendum. 
In June 2006 two companies, ESP Solutions and 
TetraData Corporation, were chosen to create the 
warehouse. 

The goal of the data warehouse is to let educators 
base their decisions on real assessment and stu- 
dent data rather than hunches. Education leaders 
want officials to “be able to back up actual deci- 
sions with actual data . . . [and] then to monitor [a] 
decision along the way to see if it is . . . paying off’ 
(interview, December 3, 2007). The warehouse is 
designed for use by individuals at all levels of the 
state education system, from district officials to 
building administrators to classroom teachers. 

The data warehouse initiative is run and supported 
by many state education department offices. The 
Office of Network and Information Services, 
assigned to build and maintain the warehouse. 
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now coordinates the work of the outside service 
providers selected to create it. The office answers 
questions from the external vendors, provides the 
data to them, and helps to coordinate and support 
training as components are rolled out. 

Office of Assessment and Accountability staff are 
working with the warehouse project to verify the 
accuracy of data uploaded into the system. In 
addition, this office provides workshops through- 
out the year to help users interpret assessment 
data. The workshops are timed to coincide with 
the release of the data from the state. In addition, 
specialized workshops help districts identified as 
in need of improvement. 

The data warehouse has seven components (some 
not funded under the current project): 

• A consolidated educational information 
system portal. 

• A data analyzer application. 

• DASH, a front-end vision alignment tool for 
teachers and principles. 

• New data collection software. 

• Data collection using the Schools Interoper- 
ability Framework technology. 

• A master directory. 

• A graphic interface system. 

The warehouse is being introduced gradually, with 
components launched at different times over the 
two-year period. Each component is piloted with 
smaller groups before being introduced to all the 
districts. 

The first component, the consolidated educational 
information system portal, was released in May 
2008. The master portal for the warehouse, it will 
give all users a single logon to access all the pro- 
grams at their access level. The second component. 



the data analyzer application, also is online and 
was introduced to all districts in March 2007. The 
third component, DASH, released in fall 2007, is 
an interface with graphs and gauges on various 
data such as today’s attendance rates. The graphics 
are based on a car’s tachometer, with information 
color coded to indicate problem areas (red) and 
areas of trouble (yellow). 

Additional formally funded warehouse com- 
ponents include new data collection software 
(released in October 2007) and a component to 
automate data collection in real time (initially re- 
leased to the Providence school district in October 
2007). The Schools Interoperability Framework 
(SIF) technology used for the data collection is 
designed to help all system directories speak the 
same language, allowing a free flow of data within 
the system. Explained one department of educa- 
tion official: 

Let’s say lama district and I have a differ- 
ent computer system to track transportation. 

I have a different computer system for the 
lunch program. I have a different computer 
system for the library system and a different 
one for the student information system. Hone 
of these systems talk to each other. So, the 
idea is, let’s say, the student information sys- 
tem is speaking French and the library system 
is speaking Spanish, SIF translates everything 
into English and then it’s kind of like the gate- 
way for communication. So, when a student 
enrolls into the student information system, 
a packet of information will be automatically 
sent to the library system saying this is a new 
student, enroll this student into the library. 

So, the idea again is to save time, data entry, 
and that type of thing. (Interview, Decem- 
ber 3, 2007) 

Two additional components connected to the data 
warehouse were not funded under the initiative at 
the time of the study. The first, a master directory 
released in July 2007, stores directory informa- 
tion about districts and schools. The second is a 
graphic interface system that would display all 
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system information graphically. Coordinated by 
the SIF technology, the interface system would 
automatically update access to all warehouse 
systems. Its release date was still to be determined 
at the time of the study. 

Data in the warehouse will include student in- 
formation, such as enrollment, discipline, special 
education, and English proficiency data; statewide 
and local assessment data; financial information; 
and teacher certification information, with other 
information about teachers that can be linked to 
their classes and students. 

Many goals are envisioned for the warehouse. The 
underlying one is to give educators the data they 
need to make the best decisions for their students, 
looking at programs and examining which have 
had better results. The warehouse will allow edu- 
cators to examine treatment and effect — always an 
important goal and one that is especially crucial 
now, during a state of fiscal crisis. Tight resources 
make it critical that educators make informed 
program decisions; good data can help them. 

The state education department has taken the lead 
in creating the warehouse. With the help of ESP So- 
lutions and TetraData Corporation, it is creating a 
system that can be used by anyone with an interest 
in education. The primary focus has been on estab- 
lishing the system and orienting users to compo- 
nents as they are introduced. Initially, training was 
done by the contractors creating the system, with 
emphasis on learning to use each new application. 
Current use and training were limited at the time 
of the study; however, the system is designed for 
many different people. Although access to and per- 
mitted uses of the data will vary based on a user’s 
position, everyone will have appropriate access to 
the data through a variety of available tools. 

Other state supports for data use. The Rhode 
Island Department of Elementary and Secondary 
Education created the School Accountability for 
Learning and Teaching (SALT) survey and process 
in 1998. The SALT process is a school-centered 
cycle of inquiry designed to improve Rhode Island 



school and student performance. The survey is 
administered by the National Center on Public 
Education and Social Policy at the University of 
Rhode Island. The university provides reports for 
every public school in the state and offers materi- 
als and technical assistance to help schools use the 
survey data to improve student achievement. 

Challenges to implementation. The state has 
encountered numerous challenges — from capac- 
ity to time to funding — as it works to implement 
the warehouse. Funding to support training for 
users at different levels is limited. Also limited 
are the personnel who support both the creation 
of the warehouse and its ongoing maintenance 
(training, checking data, and so forth). Windows 
of time when all the right individuals can attend 
an appropriate training session have been very 
limited, the system being designed for users with 
different levels of exposure to data use. In addi- 
tion, this project faces a challenge in coordinating 
its technical and analytic components: people with 
a technical understanding of the system are not 
always familiar with the analyses that will be run 
with the information. In contrast, the users of the 
system are not always familiar with the system’s 
technology but do know what kinds of analyses 
they want to conduct. This could lead to a techno- 
logically sophisticated system not meeting users’ 
analytical needs. 

Additional resources. These include: 

• The Rhode Island Department of Elementary 
and Secondary Education Office of Network 
and Information Services data warehouse 
web site at www.ride.ri.gov/onis/DW/ 
DataWarehouse.aspx. 

• The SALT survey web site at www.ride.ri.gov/ 
PSI/salt/default.aspx and the “SALT Survey 
Databook Overview” at www.nksd.net/ 
SALT/2008/overview.pdf/. 

• University of Rhode Island National Center on 
Public Education and Social Policy “HiPlaces 
School Improvement Planning and Monitoring 
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Process: Instruction Matrix” at www.ncpe.uri. unique response to the issue of schools failing to 

edu/research/matrix-instruction.asp. make adequate yearly progress. 



Vermont 

Vermont has a history of collecting and using data 
that predates the NCLB Act. The Vermont Depart- 
ment of Education has focused its data work on 
the human side of data — increasing the capacity 
of schools to collaboratively examine data and 
make instructional decisions. The state has a data 
warehouse and provides professional development 
on data analysis, contracting this work to partner 
organizations. 

State context. Vermont has the highest percent- 
age of rural residents of any state. Schools are 
decentralized and the state has many small 
schools and single-school districts. Districts are 
organized into supervisory unions, each under 
a single superintendent. Vermont’s emphasis on 
local control has resulted in a recent education 
accountability law (Act 60, the Equal Education 
Opportunity Act) specifying that the state will not 
take over schools that fail to make adequate yearly 
progress. However, the state education commis- 
sioner determines whether districts are making 
progress in improving student performance and 
holds those not making sufficient adequate yearly 
progress accountable for using data and reporting 
progress. 

Vermont has joined with New Hampshire and 
Rhode Island to develop the NECAP, administered 
each fall since 2005 in grades 3-8 and in fall 2007, 
for the first time, in grade 11. 

Overview of the initiative. The Vermont Depart- 
ment of Education sees two components in data 
use: the technical side, or the data and its analysis, 
and the human side, or the importance of teams 
combining different perspectives to learn from the 
data. This dual emphasis is reflected in require- 
ments that principals (as of 2007) and teachers 
(partly in 2008, fully as of 2009) participate in 
learning communities focused on using data 
to improve achievement. This requirement is a 



Because Vermont’s history with data use predates 
the NCLB Act, it has traditionally focused on using 
data to improve instruction rather than to meet 
accountability requirements. The state assessment 
coordinator described Vermont’s vision for the use 
of data in Vermont’s classrooms: “I think teachers 
ought to be able to tell you at any moment in time 
where each kid’s at” (interview, October 26, 2007). 

An initiative launched in the 2007/08 school year 
requires that Vermont schools identified as in need 
of improvement provide data-based evidence on 
the effect of their school improvement efforts. The 
state education commissioner’s Required Actions 
(16 Vermont State Law 165[b]) codify some of the 
responsibilities for low-performing schools and 
make these uniform statewide. All identified low- 
performing schools must: 

• Identify measures to track student progress, 
with particular focus on the groups and 
content areas for which the school is identified 
(this triggers required implementation and 
monitoring of local assessment systems). 

• Develop a continuum of support for student 
learning. 

• Report to the state at the middle and end 
of the academic year, using data to show 
progress. 

• Have principals participate in principal learn- 
ing communities. 

• Establish one teacher learning community 
during 2008/09. 

• Have all teachers in teacher learning commu- 
nities beginning in 2009/10. 

According to the state assessment coordinator, 
these requirements are to ensure that school lead- 
ers can show the state education commissioner 
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that “they know whether kids are meeting 
standards. There are lots of ways to do this; we 
just want to know you’re doing something. The 
Commissioner is saying, ‘Tell me how you know 
where your kids are.’” Through this directive, the 
commissioner ensures that identified schools use 
data to guide their school improvement efforts. 

In addition to this initiative focused on low- 
performing schools, Vermont is also working to 
expand the functions and uses of the state’s data 
warehouse so that more districts can access it and 
better use data. The warehouse restructured its 
pricing to let smaller districts, with fewer re- 
sources, afford the services. 

Schools are required to submit reports directly to 
the state education commissioner. The assessment 
coordinator manages a range of data and assess- 
ment projects within the department. Three school 
support coordinators within the department lead 
the initiative for low-performing schools. The 
Vermont Data Consortium, a nonprofit group that 
manages the data warehouse, has close ties to the 
department; one department employee worked 
with the consortium to develop queries to analyze 
data. 

In Vermont data-driven decisionmaking is sup- 
ported by the data warehouse, which stores data 
and facilitates data analysis, and by policies that 
provide incentives and accountability for data use. 
The Vermont Department of Education focuses 
more of its resources and personnel on supporting 
low-performing schools than on supporting data 
use by all schools, which is handled by partner or- 
ganizations such as the Vermont Data Consortium 
and educational service agencies. 

The Vermont Data Warehouse is a data storage 
and retrieval system, using software from Tetra- 
Data, that allows participating districts to access 
state assessment data (reported through spread- 
sheets) and generate queries to analyze those 
data. According to the consortium, the goal of the 
warehouse is to “ensure that Vermont supervisory 
unions/districts have cost-effective, timely, and 



accurate student and educational data available to 
address continuous school improvement as well 
as state and federal accountability and reporting 
requirements” (Vermont Data Consortium 2008). 
The consortium has worked with the department 
to create a Common District Model that includes 
data on students, teachers, courses, class sec- 
tions, student schedules, course grades, and local 
assessments. The model will eventually expand to 
contain data on attendance, discipline, programs, 
and interventions; standards-based report cards; 
and more. In addition, the data warehouse soft- 
ware can use data stored in any student informa- 
tion system to generate reports. Data from NECAP, 
the Vermont Development Reading Assessment, 
and local assessments are also housed in the 
warehouse. Data are collected by schools, districts, 
and supervisory unions. About half of Vermont 
districts were participating in the warehouse at the 
time of the study. 

The data warehouse allows teachers (at the class- 
room level), administrators (at the school level), 
and central office personnel (at the district level) 
to query the data through different filters. One can 
analyze how students performed on assessments, 
how different subgroups performed (in Vermont 
the most critical achievement gap is between stu- 
dents of low and higher socioeconomic status), and 
how they compare with other students in the state. 

The state department of education provides tech- 
nical assistance and, due to its small size, limited 
professional development in data use. The six 
educational service agencies (Vermont’s regional 
education service providers) can provide more 
extensive professional development, including in 
data-driven decisionmaking; however, given the 
agencies’ decentralized model, the department has 
little control over the content and quality of the 
professional development they provide. 

For schools falling under the Required Actions 
legislation, support is available from principal 
learning communities in each of the state’s four re- 
gions, as well as with technical assistance on data 
use. Unique to Vermont’s accountability efforts. 
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such communities bring together principals of 
schools failing to make adequate yearly progress to 
share perspectives and reflect on school improve- 
ment efforts. 

Challenges to implementation. Vermont’s history 
of using data, including its focus on assessment 
in the decade before the NCLB Act, has created 
challenges to adding an accountability component 
to its work on data-driven decisionmaking. As 
the assessment coordinator said, “The notion of 
failing schools came later and unwillingly to us.” 
Vermont has opted for the least punitive treat- 
ment for schools failing to meet adequate yearly 
progress requirements, focusing on support rather 
than sanctions. Other means, such as the Required 
Actions legislation, address accountability. 

An additional challenge is that not all districts 
are members of the Vermont Data Consortium, 
the most powerful tool available for data-driven 
decisionmaking. 

Next steps. The state department of education 
is providing increasing support and guidance 
for schools under corrective action. Besides the 
requirements detailed above, the department has 
been adding required teacher learning com- 
munities for all faculty in schools identified as 
low performing to be phased in by 2010. These 
communities will use data to develop strategies for 
improving student achievement. 

In addition, the department and the Vermont Data 
Consortium are exploring ways to make the data 
warehouse more accessible to more schools and 
districts through a new fee structure, with lower 
costs for small districts. 

Additional resources. These include the Vermont 
Data Consortium web site at http://vermontdata.org. 



The Virgin Islands 

The Virgin Islands Department of Education is 
in the early stages of crafting a policy to support 
data-driven decisionmaking in its districts and 



schools. The Virgin Islands Territorial Assess- 
ment of Learning (VITAL) has been in place since 
2004/05, following a five-year moratorium on 
standardized testing. After territory wide testing 
resumed with VITAL in 2004, the assessment 
was taken only by students in grades 5, 7, and 11 
in each of three successive administrations. The 
department was only beginning to develop a longi- 
tudinal data system at the time of the study. 

Context. The Virgin Islands, consisting of St. 

Croix, St. John, and St. Thomas, has a population 
of only 120,000 (primarily on St. Thomas and 
St. Croix). Although the population is mostly of 
African descent, St. Croix has a large Hispanic 
community (mainly of Puerto Rican descent), and 
St. Thomas has a significant community of Danish 
and Lrench descent. In addition, there are U.S. 
mainlanders as well as people from the Dominican 
Republic, West Bank and Gaza, and many other 
locations. All public school students qualify for 
free or reduced-price lunch. 

The education department oversees two school 
districts, with 34 schools and about 18,500 stu- 
dents. In 2003 all 34 schools were deemed at risk 
and ranked in the bottom third of all schools in 
the United States and territories. The National As- 
sessment of Educational Progress results for 2000 
show that the share of grade 4 students perform- 
ing at or above proficient level was 4 percent, 
compared with 28 percent nationally. 

Overview of the initiative. The Virgin Islands De- 
partment of Education’s director of planning and 
evaluation reported that, at the time of the study, 
it had only recently begun to create a longitudinal 
data system and was just starting to craft a data- 
driven decisionmaking initiative: “In the absence of 
formal policy, what we have been doing is using the 
NCLB [Act] as a motivator to encourage schools to 
use data” (interview October 26, 2007). Lor over a 
year the department did not have a permanent con- 
firmed commissioner of education.^^ In the absence 
of a commissioner, the director of planning and 
evaluation was setting the direction for data-driven 
decisionmaking. One of the primary motivations 



54 



STATE EDUCATION AGENCIES IN THE NORTHEAST & ISLANDS REGION AND DATA-DRIVEN DECISIONMAKING 



for helping districts and schools use data was to 
help them understand why their schools were not 
making adequate yearly progress. 

The education department contracted with the 
education information company Pearson School 
Systems to support it in developing and adminis- 
tering its assessment and longitudinal data system. 
Pearson was also contracted to hold workshops 
for principals twice a year — to teach them how 
to query data in the school administration and 
student information system database, how to enter 
data into spreadsheets, and how to pull out the 
types of information they need. 

Data management staff from the research and 
evaluation office also meet with the schools and 
stress the importance of data, helping them un- 
derstand the significance of the data for making 
adequate yearly progress. According to the direc- 
tor of planning and evaluation: 

One of the things we have done is share with 
them where their challenges are, based on 
the data in regards to their school’s perfor- 
mance. . . . We found . . . in our last report 
card that participation rate and attendance 
were two of the big areas why schools didn’t 
make [adequate yearly progress]. We found 
in the [2006/07] school year it is no longer a 
significant factor for missing [adequate yearly 
progress]. Now they are working on address- 
ing the proficiency achievement attainment of 
the subgroups. (Interview, October 26, 2007) 



Next steps. The Virgin Islands has only recently 
begun using data for decisionmaking. At the time 
of the study its plans were evolving; it planned to 
incorporate a comprehensive data warehouse and 
longitudinal system that would include achieve- 
ment data, student information, and staffing in- 
formation. In the words of the director of planning 
and evaluation: 

One of our plans is to incorporate a compre- 
hensive data warehouse and longitudinal 
system [in which] data is gathered from 
the schools and uploaded to the state. . . . 

We will be able to not just look at student 
achievement data and see whether or not all 
the teachers are highly qualified but . . . also 
be able to electronically cross-check those 
data to other sources and facilities: what’s 
happening nutritionally in the school lunch 
area, and the like. We are in the planning 
stage; we are developing a proposal to see 
what it will cost. We are moving forward 
with that. We are moving forward with 
trying to implement another level of data- 
driven decisionmaking. (Interview, October 
26, 2007) 

Additional resources. These include: 

• The Virgin Islands Department of Education 
web site at www.doe.vi. 

• The Pearson School Systems web site at www. 
pearsonschool.com. 
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APPENDIX E 

CATALOGUE OF SERVICE PROVIDERS 

This appendix provides a brief profile of nine 
Northeast and Islands Region service providers. It 
includes providers mentioned by state education 
agency respondents in interviews as the lead pro- 
vider in the state; it is not an exhaustive catalogue 
of all providers working in the Northeast and 
Islands Region. 



Cognos 

Used by more than 23,000 organizations in 
industries ranging from retail to defense, Cognos 
business intelligence software is an integrated 
performance management software by IBM that 
is relatively new to education. It is used to ana- 
lyze and report on student performance data, as 
mandated by the NCLB Act. Rather than a simple 
repository for data, it integrates and aggregates 
data to multidimensionally analyze student 
performance by various factors. Parents, teachers, 
principals, and district and state administrators 
can access and manipulate data from the software 
according to their needs. 

The Massachusetts Department of Elementary 
and Secondary Education has partnered with 
Cognos Corporation to create and manage a 
statewide data warehouse. Local districts are key 
to creating an accurate and consistent database; 
at the time of the study a pilot group of districts 
was participating in the data warehouse project. 
Project participants must complete training, 
which the department offers to end users, report 
authors, and data leaders. The “long-term goal,” 
explains one official at the department of educa- 
tion, “is to provide every district and school with 
the ability to easily query and analyze its organi- 
zation’s state-maintained data [Student Informa- 
tion Management System, Massachusetts Com- 
prehensive Assessment System, Massachusetts 
English Proficiency Assessment, and educator 
data], and to provide districts with the option to 
load and analyze their own data.” 



Connecticut Regional Educational Service Centers Alliance 

The Connecticut Regional Educational Service 
Centers (RESC) Alliance comprises six educational 
service centers that provide responsive programs 
and services to Connecticut schools and districts 
in a cost-effective manner. As part of the Con- 
necticut State Department of Education’s Connect- 
icut Accountability for Learning Initiative, one 
of its main services is professional development 
and technical assistance in data-driven decision- 
making for low-performing schools. Training for 
teachers and school administrators focuses on 
the data-driven decisionmaking process and how 
to build and sustain this process in district- and 
school-level data teams. The training is based on 
a “train the trainer” model in which trainees then 
become school leaders of data-driven decision- 
making professional development. Districts that 
request it receive follow-up technical assistance 
based on their unique data-driven decisionmaking 
needs. In addition, the alliance provides a variety 
of customized work to schools and districts, such 
as spreadsheet and database training and custom 
report development. 

The RESC Alliance maintained the Connecticut 
Data Warehouse, powered by TetraData’s EASE-e 
Data Analyzer, and provides technical assistance 
and training on understanding and using the data 
warehouse and the data analysis applications asso- 
ciated with it. At one point, as many as 76 districts 
of the 169 districts in Connecticut were receiving 
some type of training and professional develop- 
ment pertaining to the data warehouse. Because of 
a lack of funding and capacity at the district level, 
however, the data warehouse was not used and 
ceased functioning at the end of June 2008. 



ESP Solutions Group 

ESP Solutions Group provides data systems and 
psychometrics in K-12 education to federal, state, 
and local education agencies. Led by a manage- 
ment team with extensive experience in technol- 
ogy and education, it pioneered the concept of 
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data-driven decisionmaking in the 1970s. ESP Interoperability Framework to automate verti- 

offers consulting and services in data manage- cal reporting of data. In progress, 

ment, data collection and exchange, data analysis, 

and data reporting. Its products include the State • Vermont — Client: state education agency. 

Report Manager web tool, publications and re- Consulting work on state’s information system 

ports, online data dictionaries, disaster prevention design (1998). 

and recovery plans, and electronic transcript tools. 

Measured Progress 



While leading the U.S. Department of Education’s 
Performance-Based Data Management Initiative 
(now called Education Data Exchange Network), 
ESP worked with all states and territories. Past 
and present work with the Northeast and Islands 
Region states includes: 

• Connecticut— Client: state education agency. 
Developing a statewide longitudinal educa- 
tional data system using ESP’s State Report 
Manager web tool and other dissemination 
tools. In progress. 

• Maine — Client: state education agency. Tak- 
ing data inventory using DataSpecs and ESP 
metadata inventory tool. In progress. 

• Massachusetts — Client: Boston Public 
Schools. Past services for the Massachusetts 
Department of Education include making an 
applications inventory for the chief informa- 
tion officer (2006); consulting on statewide 
student identifiers, confidentiality, and data 
standards (2002); building a statewide student 
identification system (1998); and developing a 
data dictionary (1996). 

• New Hampshire — Client: state education 
agency. Documenting state’s i4see data man- 
agement policies and procedures for student 
data access and data verification. In progress. 
Dropout data collection (1999). 

• New York — Client: state education agency. 
Consulting on student identifiers (2002). 

• Rhode Island — Client: state education agency. 
Designing and implementing a data ware- 
house; implementing a statewide Schools 



Measured Progress, an independent nonprofit 
organization, develops, implements, and oper- 
ates education assessments for grades K-12 and 
provides related professional development. It 
also houses two research centers: one focuses on 
theoretical and applied psychometrics; the other, 
educational assessments. Consulting services in 
various areas are also offered. 

Its K-12 assessment services division covers all 
aspects of large-scale, alternative, classroom, 
and online assessments and counts the following 
Northeast and Islands Region states as former and 
current clients: 

• Connecticut and New York — alternative 
assessments. 

• Maine — Maine Educational Assessment. 

• Massachusetts — Massachusetts Compre- 
hensive Assessment System; Massachusetts 
English Proficiency Assessment. 

• New Hampshire — New Hampshire Educa- 
tion Improvement Assessment Program, from 
1992-2006. 

• New Hampshire, Rhode Island, and Ver- 
mont — New England Common Assessment 
Program. 

Focusing on assessment literacy and effective as- 
sessment strategies, the professional development 
services division has delivered customized work- 
shops in districts and schools across the North- 
east and Islands Region. It also produces a guide 
to interpreting the state reports so that teachers 
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and administrators can use assessment data to 
inform, drive, modify, and improve instruction 
in the classroom. At the time of the study a new 
product — i-analyze — that would serve as a report- 
ing portal with filtering options for schools was in 
development. 



National Center for the Improvement 
of Educational Assessment 

The National Center for the Improvement of 
Educational Assessment, Inc., or the Center for 
Assessment, is a nonprofit organization founded 
in 1998 to address the changes in accountability 
and assessment under way across the country. 

The center works with state and other education 
agencies to design and implement effective assess- 
ment and accountability policies and programs. It 
provides services in five areas: technical advisory 
committees, assessment systems, accountability 
systems, other consulting, and Reidy Interactive 
Lecture Series conferences. Since its founding, it 
has worked in 25 states; at the time of the study 
it was working with 11 state departments of 
education and one nonprofit organization. In the 
Northeast and Islands Region it was working with 
the Massachusetts and Vermont state departments 
of education: 

• Massachusetts. The Massachusetts Depart- 
ment of Elementary and Secondary Educa- 
tion was creating an accountability system 
that is part of the Education Reform Law and 
that validates its psychometric measures and 
design scheme. The Center for Assessment 
helped to review the proposal for this system 
and its education and policy issues. 

• Vermont. The Vermont Department of Educa- 
tion needed an accountability system that 
could meet its unique characteristics: many 
schools with small student populations and 

a history of local control. The state wanted a 
system that could house state tests and other 
assessment as well as incorporate locally 
determined data into a reliable measure of 
school achievement and progress. The center 



helped the department develop an account- 
ability plan based on its specific needs. 



New York Board of Cooperative Educational Services 

The Board of Cooperative Educational Services 
(BOCES), created by the New York State Legisla- 
ture in 1948, consists of regional support networks 
that provide services to districts in need of help, 
including technical assistance and technology sup- 
port. Participating school districts receive services 
through a cooperative service agreement, though 
some services are free. BOCES provides data- 
driven decisionmaking services based on regional 
and district needs. Many low-performing schools 
seek BOCES help to use data to improve teaching 
and learning. 

Housed in the 37 BOCES in the state are 12 
statewide regional information centers that give 
districts access to instructional and administra- 
tive technology services as well as training and 
support for teachers, students, and administrators. 
They store and process all the data required for 
the state and provide districts with customized 
professional development on data management 
and analysis. 



Pearson School Systems 

Pearson School Systems is a part of the media 
giant Pearson that provides education, business 
information, and publishing services. Creating 
innovative education technology for 40 years, 
Pearson has the largest market share in student 
information systems solutions — with 6,500 school 
districts and 130,000 schools serving 22 million 
students using its products. In addition to its 
award-winning software, Pearson provides com- 
prehensive consulting, implementation services, 
and project management. 

Some of Pearson School Systems’ most popular 
data products include: 

• Chancery SMS. A solution for growing urban 
districts seeking specific implementation 
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and customization services, streamlined 
operations, and organizational process re- 
engineering. 

• Pearson Inform. A data analysis and deci- 
sion support tool for K-12 schools. Performs 
comparative and longitudinal analyses from 
student to district level. Analyzes data from 
multiple sources. 

• PowerSchool Premier. A fully integrated, web- 
based, cross-platform student information 
system. It is a real-time, intuitive user-friendly 
communication tool. 

• SASl. A flexible, comprehensive student 
information management system that allows 
teachers and administrators to monitor, track, 
and report on student data and progress 
(through SASI Gradebook, Parent Access, and 
Classroom). Included in the system is a com- 
munication tool that allows parents to track 
their children’s academic progress online. Its 
focus is on day-to-day operations. 

Performance Pathways 

A technology company formed in 2005 by the 
merger of Technology Pathways Inc. with Alter- 
Net Performance, Performance Pathways Inc. 
focuses on data analysis and a data-informed 
culture in education. It believes that districts need 
to have curriculum-driven assessment and that 
curriculum and assessment data should drive 
decisionmaking. 

Performance Pathways develops and provides cur- 
riculum and assessment solutions for the PreK-12 
environment. Its three integrated applications — 
Assessment Builder, Performance Tracker, and 
TechPaths Curriculum Mapping System — are de- 
signed to help teachers and administrators make 
informed and timely decisions about instruction: 

• Assessment Builder. Designs, scores, and 
analyzes local assessments. Designed to assess 
benchmark skills to predict how students will 



perform on the main state test. Instruction 
can then focus on the identified area of need. 

• Performance Tracker. An assessment manage- 
ment tool that collects assessment data and 
writes reports. It is web based (accessible) 
and integrates with other popular software. 
Curriculum and assessments are correlated to 
state and district standards. 

• TechPaths. A curriculum mapping system 
developed by Dr. Heidi Hayes Jacobs. It uses 
assessment data and links to curriculum data. 

More than 55,000 teachers from 500 schools in 
47 states were using products from Performance 
Pathways at the time of the study. Most North- 
east and Islands Region states have districts that 
employ Performance Pathways: 

• New Hampshire — As part of its Follow the 
Child initiative, the state has provided Per- 
formance Tracker to all public schools since 
spring 2007. The system currently includes 
assessment data from the New England Com- 
mon Assessment Program and Northwest 
Evaluation Association, but more indicators 
will be added later. It is geared primarily 
toward teachers and principals but will ulti- 
mately be available to students. 

• New York BOCES — All BOCES districts can 
use the applications. Districts can obtain 
professional development directly from 
the BOCES or become certified and train 
themselves. 

• Connecticut, Massachusetts, and Rhode 
Island — Client list of districts is increasing. 



TetraData 

Founded in 1997, TetraData became a subsid- 
iary of Follett Software Company in 2006. Its 
products are data-driven performance solutions 
for grades K-12. Districts can come aboard any 
time in implementing their data system; they are 
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empowered to develop improvement plans as well 
as monitor, evaluate, and change their strategies 
along the way. 

TetraData Analysis Suite, TetraData DASH, and 
TetraData Warehouse and Central Data Store are a 
suite of solutions. A combination of these products 
and services is packaged to meet each district’s 
needs. 

• Analysis Suite. Analyzes and monitors data 
factors, including longitudinal analyses; mea- 
sures program efficacy; tracks disaggregated 
No Child Left Behind subgroups. Intended for 
use at the district level. 

• DASH. Front-end vision alignment tool for the 
warehouse used by teachers and principals. A 
communications tool, DASH allows the dis- 
trict to quickly analyze data and disseminate 
them to principals and teachers. 



• Warehouse and Central Data Store. Consoli- 
dates databases into a central information 
source. 

In addition to the aforementioned products, Tetra- 
Data provides data management services, profes- 
sional development and training, hosting facility 
services, and customer service that includes a 
project manager assigned to each client district. 

More than 600 K-12 education institutions na- 
tionwide, serving more than 2.3 million students, 
use TetraData products and services in one form 
or another. Its clients are typically districts with 
an average of 15,000-30,000 students; the cost of 
the system often precludes smaller districts from 
purchasing it. A typical contract is renewable and 
lasts for three years. In the Northeast and Islands 
Region New York City, Rhode Island, Vermont, 
and at least one district in Connecticut are clients 
of TetraData. 
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APPENDIX F 

SERVICE PROVIDER PROFILES 

Three service providers were selected for profil- 
ing. Providers had to meet the following criteria: 
work with multiple jurisdictions in the region, 
have been in existence for two or more years, and 
have offered these services for two or more years. 
In addition, organizations had to offer professional 
development in using data to inform teaching and 
learning, not just support on the other compo- 
nents. This appendix contains in-depth profiles of 
those three service providers — the Connecticut 
Regional Educational Service Centers (RESC) 
Alliance, Measured Progress, and Performance 
Pathways. 



Connecticut Regional Educational Service Centers Alliance 

The Connecticut Regional Educational Ser- 
vice Centers Alliance is a consortium of the six 
regional educational service centers in the state. 
Created under state statute, the centers come 
together to share common problems and solutions 
and determine cost-effective ways of meeting the 
needs of public school districts through various 
programs and services. The RESC Alliance works 
with Connecticut districts that request its services, 
though it has also worked in other Northeast and 
Islands Region states (Massachusetts and Rhode 
Island). Service areas offered include technology, 
assessment, curriculum, data warehousing, and 
data-driven decisionmaking, among others. The 
centers are funded partially by the state educa- 
tion department, partially by revenues from their 
services, and partially by federal grant money. 

Overview. The Connecticut RESC Alliance was 
formed in 2000 after the regional educational 
service centers recognized the strong need for 
data warehousing in districts. This was before the 
Connecticut State Department of Education estab- 
lished the Connecticut Accountability for Learn- 
ing Initiative (CALI), which provides professional 
development in data-driven decisionmaking and 
technical assistance to low-performing districts, 
among other services (see the Connecticut profile 



in appendix D). At the time, the centers were 
unsure of when the education department would 
begin to develop an approach to data warehousing, 
so they formed the Connecticut RESC Alliance 
and contracted a vendor to provide districts with 
data warehousing along with technical assistance 
so they could use the data to make informed edu- 
cation decisions. 

Based on the recommendation of Victoria Bern- 
hardt, an expert in the data-driven decision- 
making field, TetraData was contracted to provide 
data warehousing to districts. RESC members had 
worked with Bernhardt and valued her work in 
data-driven decisionmaking. The Connecticut data 
warehouse was powered by TetraData’s EASE-e 
Data Analyzer software program.^^ In Connecticut 
76 of its 169 districts have received some type of 
training and professional development in data 
warehousing. 

The interviewees from the RESC Alliance empha- 
sized that work on data-driven decisionmaking 
should focus on data warehousing and school 
improvement planning together, be a collaborative 
process, and involve multiple methods of collect- 
ing data. 

RESC Alliance members suggested that districts 
should not only collect, house, and analyze their 
data to meet their individual needs but also do this 
with a focus on the development and monitoring 
of school improvement plans. EASTCONN, one 
of the regional educational centers, worked with 
the Hampshire Collaborative in western Massa- 
chusetts to help it develop its data warehouse and 
craft school improvement plans that would involve 
using data. One interviewee from the RESC 
Alliance described the importance of focusing si- 
multaneously on both data and data use in school 
improvement in working with districts: 

I went out and started working with [schools] 
initially just focused around data warehous- 
ing, but what I knew at that point from what 
we had tried to do in Connecticut was [how 
important it was] to really get them to focus 
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on school improvement and not just data 
warehousing. I mean you can buy a data 
warehouse and that’s fine, but if you don’t 
think about the other side of this, which is 
what you’re going to do with these data, and 
begin to develop these things down parallel 
roads, then you’re going to build a wonderful 
warehouse and . . . what we found in Con- 
necticut is that we built this great tool, but 
nobody knew what to do with it. And you 
know, for those districts in Connecticut that 
didn’t know what to do with it, they failed. 
(Interview, February 6, 2008) 

The RESC Alliance also views data-driven 
decisionmaking as a collaborative process, as one 
interviewee described: 

It’s about conducting some root cause analy- 
ses so that you’re getting to some of those 
underlying causes. I’m a firm believer that 
you can’t get to the root cause in education 
because the variables are so many and so 
complex, but you’ve got to [make] . . . your 
best attempt at getting to some root causes. 

And then setting some . . . goals and develop- 
ing some real, clear action plans. Strategies 
are fine, but the missing piece in some of this 
is having a real plan to implement those strat- 
egies. It’s completing the loop here so that . . . 
we’ve got monitoring plans in place, we’ve got 
action plans, who is responsible, all of those 
kinds of things. You know, it’s almost a mis- 
nomer to call it data-driven decisionmaking 
because you’re doing more than making a 
decision. (Interview, February 6, 2008) 

Interviewees also noted the importance of having 
multiple methods of collecting data if data-driven 
decisionmaking is to be effective. Due to pressures 
to meet adequate yearly progress targets, districts 
and schools have made collecting and examin- 
ing state assessment data from the Connecticut 
Academic Performance Test and the Connecticut 
Mastery Test a priority. The RESC Alliance recog- 
nizes that other data must also be used to make 
improvements and also that students should have 



classroom assessments to make more real-time 
decisions on instruction.'^ This vision for data- 
driven decisionmaking work is integrated with the 
services the RESC Alliance and CALI provide to 
Connecticut districts and schools. 

Clients. The RESC Alliance works with any district 
in Connecticut that requests services and has also 
worked in other Northeast and Islands Region 
states (Massachusetts and Rhode Island). The 
data-driven decisionmaking training it delivers 
through CALI had been offered for free only to 
districts in corrective action, and for a fee to other 
districts. At the time of the interviews, the RESC 
Alliance was looking for funding to provide ser- 
vices free of charge to all interested districts. 

Services. Along with technical assistance and 
training on understanding and using the data 
warehouse and the associated data analysis 
applications, the RESC Alliance also provides 
districts and schools with a continuum of services, 
including forming data teams and guiding school 
improvement planning. Customization is also 
available to schools or districts in, say, compre- 
hensive data management services and assessment 
mapping. 

RESC Alliance is involved in professional devel- 
opment services that are part of the Connecticut 
State Department of Education’s CALI, which 
aims to advance learning for all students, par- 
ticularly those from the 12 districts in corrective 
action. Initiated in 2004, CALI provides support 
to districts and schools in the following areas: 
data-driven decisionmaking data teams, mak- 
ing standards work, effective teaching strategies, 
common formative assessments, school climate 
improvement to support academic achievement, 
and accountability in district and school improve- 
ment planning. The professional development and 
technical assistance are offered to Title I districts 
and schools that are identified as in need of 
improvement. But ineligible districts and schools 
can participate for a fee. for those districts that 
choose to participate, four days per year are set 
aside for professional development; the technical 
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assistance is offered throughout the year as follow- 
up support. 

The data-driven decisionmaking component of 
CALI aims to reduce achievement gaps by pro- 
viding districts with professional development 
services and technical assistance on using data to 
impact pedagogy and learning and, by extension, 
improve student achievement. The hope is that 
these services will create a culture of data-driven 
decisionmaking in schools and districts where 
data teams of school administrators and teachers 
are examining multiple kinds of data in an ongo- 
ing format. 

These services are based on the work of data- 
driven decisionmaking experts, including Doug 
Reeves (Reeves 2000, 2001), Nancy Love (Love 
et al. 2008; Love 2002), and Victoria Bernhardt 
(Bernhardt 1998). Reeves centers his approach to 
data-driven decisionmaking and other school im- 
provement planning efforts (such as using effective 
teaching strategies and coaching) around a holistic 
accountability system where districts, schools, and 
communities are accountable for what happens 
within the system. The system must be collabora- 
tive and yield identifiable results based on set 
goals for each level of stakeholder. This process 
of collaboration provides a vehicle for examin- 
ing data and student work that leads to changes 
in teacher instruction and student learning. In 
addition. Love and Bernhardt were the first to ap- 
proach data-driven decisionmaking on a practical 
level, and their work focuses on providing techni- 
cal assistance to schools and districts and around 
establishing a cycle of inquiry and data-driven 
decisionmaking. The trainers draw from the work 
of these experts to design their professional devel- 
opment sessions. 

RESC Alliance data-driven decisionmaking 
training. Two of the regional educational service 
centers provide low-performing districts with 
professional development training in data-driven 
decisionmaking and data teams, and they have 
plans to deliver the common formative assess- 
ments training in the near future. The technical 



assistance follow-up to the professional develop- 
ment is also provided by the regional educational 
service centers— though not by the data-driven 
decisionmaking professional development trainers 
themselves — and the State Education Resource 
Center. 

The RESC Alliance trainers were initially trained 
by consultants from the Leadership and Learn- 
ing Center. These trainers could then train others, 
as well as provide on-site support, in the various 
modules that make up the organization’s approach 
to school improvement. The interviewees were 
trained in data-driven decisionmaking and soon 
will be trained in common formative assessments 
in order to provide services in it. The individuals 
who provide the basic data-driven decisionmaking 
training have complementary backgrounds 
that bolster the services provided. While they 
are both certified to train others in data-driven 
decisionmaking, one has a stronger background in 
technical aspects of data-driven decisionmaking, 
including collecting, housing, and managing data; 
the other is stronger in school improvement. 

Professional development services. The RESC Al- 
liance delivers a two-day basic training in data- 
driven decisionmaking to eligible districts that 
are part of CALI.^"* This training gives district 
teams the opportunity to examine data from 
their own school and apply it within the context 
of the guided training. The basic training is a 
seminar for teachers and school administrators, 
including, for example, curriculum developers, 
superintendents, and principals. Participants learn 
how to examine their own real student data using 
a five-step data-driven decisionmaking process 
(see appendix D for an overview of this process) 
and how to develop and sustain this process on 
district- and school-level data teams composed of 
fellow teachers and administrators. Through this 
process, data teams collaboratively analyze data 
and identify student strengths and weaknesses. 
Team members devise instructional strategies that 
best address these areas as well as the required 
learning standards. These strategies must be one 
of the Efiective Teaching Strategies identified in a 
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meta-analysis by Marzano, Pickering, and Pollock 
(2001, cited in Connecticut State Department of 
Education 2008). Team members are meant to 
implement their strategies, monitor them, and 
discuss them at their next meeting. A variety of 
actions are taken as a result of this process, includ- 
ing curriculum revision, program redesign, and 
funding redistribution. 

Participants bring their own classroom- or 
school-level data to analyze during the trainings. 
Attendees bring a variety of data, including state 
assessment data; data on behavioral issues, such 
as tracking the number of classroom referrals or 
office referrals; and school climate survey data. 

The data are typically housed on paper or in a 
spreadsheet. No technology analysis tools are used 
in the training. 

Technical assistance services. Technical assistance, 
based on particular district needs, is provided 
throughout the year to districts as a follow up 
to the professional development. Thus, it could 
involve doing the actual basic training described 
above, developing a common formative assess- 
ment, or even helping districts use spreadsheet 
software to examine their data. It could also 
involve helping districts learn how to provide dif- 
ferentiated instruction or examine student work. 

Challenges. A main goal of CALI training is to en- 
courage schools and districts to create a culture of 
educators using data frequently, with the ultimate 
goal of improving student achievement. However, 
there are challenges associated with getting data- 
driven decisionmaking practices institutional- 
ized in schools and districts in the state. The first 
challenge needing attention is teacher resistance to 
change. As an interviewee from the RESC Alliance 
commented: 

You continue to send a constant message, and 
at some point either the change starts to occur 
and they are still in the place they were in, or 
they start to internalize some of the message 
and make it their own. So that would be the 
first obstacle. (Interview, February 6, 2008) 



Furthermore, this is layered with the problem 
of districts making quick fixes to meet adequate 
yearly progress and ignoring the need for systemic 
change. Another interviewee stated: 

Everybody is so “deer in the headlights’’ 
because of [adequate yearly progress]. I mean, 
it is hard to institutionalize anything because 
they are all about quick fixes. They are about 
safe harbor, which . . . gets you out of the 
newspaper, but that does not fix the prob- 
lem. . . . You still have most of your kids not 
making [adequate yearly progress], but you 
made safe harbor. It’s like, “Wait a minute . . . 
there’s still something wrong with this picture 
here.” (Interview, February 6, 2008) 

Another challenge is the lack of time in a typical 
school day to conduct such practices as data team 
meetings. In addition, there are often not enough 
individuals in the district or school who have the 
ability and time to keep the data-driven decision- 
making work on track throughout the year. But 
one of the biggest challenges noted by interviewees 
from the RESC Alliance was the lack of a clear 
vision for data-driven decisionmaking in districts. 
This is because, as one interviewee noted, “With- 
out that shared vision, people don’t know what it 
is they are trying to institutionalize.” He also sug- 
gested that leadership is ultimately responsible for 
creating a vision of data-driven decisionmaking 
in a district or school and making sure that it is 
shared among administrators and teachers. A 
shared vision is often not institutionalized due to 
poor leadership, and sometimes teacher turnover. 
If there is an institutionalized vision in a par- 
ticular district and a new leader comes in with a 
completely different idea of data-driven decision- 
making and how it should be implemented, a 
“flavor of the month” trend can result, preventing 
the district from making continuing progress in 
data-driven decisionmaking. One interviewee 
suggested that when districts are interviewing for 
a new superintendent or principal, they should 
explain to him or her that a culture already exists 
and encourage that person to embrace it and carry 
it forward. However, there often is not a culture in 
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place, so it is also important to ensure that a new 
leader has the motivation and ability to create a 
vision and make sure it is shared by all. Finally, 
while districts have the infrastructure for adequate 
yearly progress reporting purposes, they have not 
created the classroom and formative assessments 
that are needed to collect additional data beyond 
demographic and state assessment data. 

Despite the challenges associated with institution- 
alizing a culture of data-driven decisionmaking in 
districts and schools in Connecticut, the RESC Al- 
liance interviewees cite their success at the school 
level in raising awareness of the need for data to 
inform education decisions. 

Additional resources. These include: 

• Connecticut RESC Alliance web site at http:// 
ctrescalliance.org. 

• Connecticut Accountability for Learning 
Initiative web site at www.sde.ct.gov/sde/cwp/ 
view.asp?a=2618andQ=321754. 



Measured Progress 

Measured Progress is a nonprofit company that 
provides large-scale and alternative assessments 
and professional development in using data to 
improve student achievement. Founded in 1983 as 
Advanced Systems in Measurement and Evalu- 
ation, Inc., the company has worked with more 
than 35 states and major districts to develop large- 
scale and alternative assessment programs. Head- 
quartered in Dover, New Hampshire, Measured 
Progress also has facilities in Colorado, Kentucky, 
and New York, with more than 400 full-time 
employees. 

Although the company has grown dramatically 
over the past 25 years, it remains focused on its 
original commitment to students and teaching 
and learning. Still leading the company, one of 
its cofounders began his career as an elementary 
school teacher; in fact, most senior staff mem- 
bers are former teachers who bring a wealth 



of experience in education and in supporting 
schools with Measured Progress. For example, 
the vice-president of client services is the former 
director of the Maine Educational Assessment 
testing program, and before that he was a class- 
room teacher and principal. As an employee of 
the state education agency in Maine, he worked 
closely with Measured Progress (then Advanced 
Systems) to develop and administer the assess- 
ment and was the original provider of the test in- 
terpretation workshops in Maine. In an interview 
he speculated that he has probably conducted test 
interpretation workshops as long as anyone in 
the field. Given its staff’s rich backgrounds in the 
field. Measured Progress is particularly qualified 
to support local educators in implementing data- 
driven decisionmaking. 

Overview. According to the company web site. 
Measured Progress embraces the motto, “It’s all 
about student learning. Period.” This was reiter- 
ated by senior staff and was evident in their de- 
scriptions of their work. One senior staff member 
commented on the problem of “data overload” 
in schools that does not lead to improved student 
learning, saying, “There’s way too much data out 
there and way too little information.” 

The company’s mission is to improve teaching 
and learning by providing assessment data that is 
customized to the needs of schools and then help- 
ing educators use data to improve student achieve- 
ment. The company web site describes Measured 
Progress as striving to provide services for educa- 
tors that meet the following characteristics: 

• Relevant. Assessment is not an end in itself; its 
purpose is to foster student growth. 

• Personal. Shared goals, values, and passions 
enable us to forge lasting relationships. 

• Focused. You and your students are at the 
heart of our mission. 

• Reliable. You know you can count on us to do 
what it takes to get the job done right. 



APPENDIX F. SERVICE PROVIDER PROFILES 



65 



• Tailored. Customization is part of who we are; 
we specialize in solutions that fit. 

• Student driven. It is all about student learning! 

Through its professional development, Measured 
Progress focuses on helping districts and schools 
develop several factors common to high-achieving 
schools, including a standards-based learning 
environment, teachers and administrators who 
understand assessments, effective use of a variety 
of strategies to gather information about student 
learning, adjustments and interventions during the 
learning process, professional learning communi- 
ties, collaborative teams that examine all aspects of 
student work to determine program effectiveness, 
and the use of multiple data sources to plan for 
improvement. The ultimate goal is to help teachers 
use data to help students. In the words of another 
employee, quoted on Measured Progress’ web site: 
“The intention of the company is, always has been, 
and always will be to impact how well teachers can 
help students to learn. It’s why we exist.” 

This sentiment was echoed in interviews with two 
senior staff. Both spoke passionately about how 
their goal was to support teachers and students. 
One noted his strong distaste for one-shot profes- 
sional development workshops that fail to build 
relationships with educators or to serve as a guide 
and consultant as educators deepen their under- 
standing of how evidence can shape classroom 
practice. 



Clients. Measured Progress serves districts and 
states around the country, developing and admin- 
istering state assessments for 13 states (table FI). 
These statewide assessment contracts generally 
include professional development on administer- 
ing the assessment and understanding and using 
assessment data; this is included in the state’s con- 
tract and provided to educators in all districts and 
schools in the state. Measured Progress also has 
professional development contracts with 3 states 
and with individual districts in 12 states. 

In the Northeast and Islands Region it serves 
seven states in various capacities. It has worked 
with Maine to develop and administer the Maine 
Educational Assessment since the test’s incep- 
tion in 1985. It also developed the Massachusetts 
Comprehensive Assessment System and works 
with New Hampshire, Rhode Island, and Vermont 
to administer the New England Comprehensive 
Assessment Program. It produces alternative as- 
sessments for Maine, Massachusetts, New Hamp- 
shire, New York, and Rhode Island and has been 
administering the Massachusetts English Profi- 
ciency Assessment since 2003. And it contracts 
directly with districts in Maine, New Hampshire, 
and Vermont to provide schools and districts with 
professional development that focuses on creating 
a data-driven culture. 

Services. Measured Progress primarily offers two 
kinds of contracts to states and districts around the 
country. It develops and administers large-scale 



TABLE FI 

Measured Progress clients, 2008 



Service provided 


State of district contracting for services 


State contracts 


Assessments 




Florida, Georgia, Kentucky, Maine, 
Massachusetts, Missouri, Montana, Nevada, 
New Flampshire, New York, Rhode Island, 
Utah, Vermont 


Professional development 


Georgia, Louisiana, Kentucky, Maine, 
Massachusetts, Michigan, Mississippi, 
Montana, New Flampshire, New Jersey, South 
Dakota, Vermont 


Louisiana, Nevada, South Dakota, 



Source; Authors' compilation. 
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and alternative assessments to help states and 
districts understand their assessment reports, 
and it provides professional development services 
to improve the capacity of educators to make 
data-driven decisions.'^ Most of these services 
are provided under multiyear contracts, allowing 
Measured Progress to tailor its assistance to client 
needs. Professional development is delivered by 
former educators, whose experience lets them focus 
on the needs of schools and classrooms. 

The professional development services provided 
by Measured Progress address relevant catego- 
ries discussed in the research on data-driven 
decisionmaking: building assessment literacy 
(Price and Koretz 2005; Center for Research 
on Evaluation, Standards, and Student Testing 
2006), developing inquiry cycles to focus data 
analysis (Boudett, City, and Murnane 2005; 
Bernhardt 1998), examining instruction (City, 
Kagle, and Teoh 2005; Love 2002), and improv- 
ing achievement (Love et al. 2008). Measured 
Progress is beginning to create data analysis tools 
(Wayman, Stringfield, and Yakimowski 2004). 
The company also houses two research centers, 
which help it stay up to date on large-scale and 
alternative assessment, formative assessment, 
and using data to improve classroom practice. 
Researchers from Measured Progress regularly 
present at state, regional, and national confer- 
ences. Measured Progress also stays current 
through its National Advisory Committee, made 
up of education experts; its partnership with 
the National Middle School Association; and its 
memberships in the Council of Chief State School 
Officers and the Council of Great City Schools. 

Large-scale assessments. Measured Progress 
develops and administers various large-scale 
assessments. Most are criterion-reference tests, 
administered to students in each grade level along 
with alternative assessments for students who 
cannot participate in the regular assessment pro- 
gram. Measured Progress also administers high 
school graduation tests for Georgia and Nevada, 
writing portfolios for Kentucky’s comprehensive 
assessment system, and a test of language arts 



proficiency for English language learner students 
(the Massachusetts English Proficiency Assess- 
ment) for Massachusetts. 

It works with the states to develop large-scale and 
alternative assessments customized to the needs of 
each state education agency — aligning assessment 
items with state content and skills standards, for 
instance. The company scores the tests and creates 
test reports for each school and district and works 
closely with the states to support districts and 
schools in interpreting the test reports. 

Professional development to support assessment 
contracts. In conjunction with the state assess- 
ments, Measured Progress provides professional 
development for local educators to help them 
make meaningful decisions using their test data. 
Supports include detailed assessment reports and 
workshops to help educators use the data to im- 
prove classroom practice and school and district 
programs. 

Because Measured Progress believes a well 
designed score report is the first step in helping 
educators interpret assessment data, it carefully 
crafts its reports to guide discussions about school 
improvement. Explains a senior staff member: 

The extent to which schools can use data from 
assessment is driven by the professional de- 
velopment they receive to some extent, but it’s 
also driven by the design of the reports. We’ve 
always tried to design reports to maximize 
information to schools. It’s the first line [for 
deciding] what information you want to com- 
municate. What do you want to be able to say 
based on the results of these tests? What infor- 
mation do you want to give schools? We focus 
around . . . three essential questions [How did 
we do? What do the data tell us about parts of 
our program? What do the data tell us about 
parts of our population (No Child Left Behind 
subgroups)?]. If I see three years in a row that 
students always do worse in geometry, then 
that’s program evaluation in its purest sense. 
(Interview, March 24, 2008) 
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Reports are tailored for each state education 
agency. There is no boilerplate test report: each 
report looks different, based on the needs and 
priorities of the state education agency. Measured 
Progress tries to make the reports complement 
and align with other initiatives in the states. They 
provide item-level information linked to state 
standards so local educators can make meaningful 
decisions about school programs and classroom 
practice within the context of their state’s account- 
ability system. In the words of one administrator 
from Measured Progress: 

The design of the reports is really where it 
starts, I think. The linking of data on in- 
dividual item results is a way of making it 
easier for teachers to grab hold of the data 
and make some sense of it. The item-level 
report has generated more conversation 
among teachers than anything we’ve ever 
produced. It gets real personal on a kid level, 
and it helps them to think about why students 
got something wrong. . . . Massachusetts 
releases 100 percent of [the test items] every 
year. Maine and NECAP [states] would if 
they could afford it. . . . When you can show a 
teacher an actual item and how all their kids 
performed on that item, it becomes a per- 
sonal thing for the teacher and that’s really 
good. (Interview, March 24, 2008) 

In interviews, senior staff at Measured Progress 
made clear their belief that Measured Progress score 
reports are a major contribution to helping local 
educators use data to make informed decisions. 

Measured Progress also offers workshops on test 
interpretation as part of its large-scale assessment 
services. These workshops focus on answering 
three important questions through the data: How 
did we do? What do the data tell us about parts of 
our program? What do the data tell us about parts 
of our population (the subgroups)? As one senior 
staff member at Measured Progress explained: 

It’s not just about training people how to 
look at reports; it’s about what you do with 



them. . . . In the workshops we try to make 
the data come alive. Schools bring their own 
reports. We walk them through each one of 
the reports and give them clues about where 
they would find data about answering. . . 
those [three] questions. (Interview, March 24, 
2008) 

Measured Progress works with the state education 
agency to design and implement the workshops, 
which typically last half a day and are offered in 
various locations in the state. Workshops targeted 
district administrators, principals, and teachers, 
who not only review their own data but are also 
expected to share the process with others at their 
schools. 

Until recently, these workshops were face-to- 
face and included little technology. As Measured 
Progress moves toward online test administration 
and score reports, workshops are beginning to 
integrate more technology in order to train users 
on the new web-based data portal. 

Professional development to support data-driven 
decisionmaking. Measured Progress offers addi- 
tional professional development, separate from its 
large-scale assessment contracts, to help educa- 
tors use data to inform decisionmaking. These 
professional development contracts are primarily 
with individual districts. One senior staff member 
noted: 

We have a variety of professional develop- 
ment offerings, from looking at data to look- 
ing at student work. It’s all loosely connected 
with assessment. . . . It’s more elective on the 
part of districts, based on their own judgment 
or readiness for a variety of things. All profes- 
sional development is customized for their 
needs. (Interview, March 24, 2008) 

Measured Progress helps districts create classroom 
assessment tailored to individual local client needs 
and aligned to state standards, so students are 
better prepared to take the high-stakes assess- 
ment. It also offers support for creating a formative 
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assessment process, which the company defines as 
adjustments and interventions during the learning 
process. 

These contracts are longer term, not isolated work- 
shops. This is part of the overall philosophy of the 
company, which urges building a relationship with 
clients to better meet their needs. The director of 
professional development described how essential 
this was to him: 

When I joined Measured Progress, it was 
doing a lot of workshops. I wanted to move 
toward longer term relationships with clients. 

It was a condition of joining the company. 
(Interview, April 10, 2008) 

Measured Progress does offer low-cost, one-day 
workshops, however, primarily as an enticement to 
establish a longer term relationship. 

Challenges. Measured Progress has been busy 
helping state education agencies scale up their 
large-scale assessments in the wake of the NCLB 
Act. Although most state education agencies 
administered statewide assessments prior to 2001, 
they now test all students every year, beginning in 
grade 3. Measured Progress staff understand and 
respond to the pressure this puts on states: 

Departments of education are being asked to 
do exponentially more with fewer resources. 
Prior to NCLB, most states tested at three 
grade levels. Overnight, they had to more 
than double the number of kids they were 
testing. And in most cases [education depart- 
ments] were doing it with the same and even 
reduced staff. And they were also trying to 
implement data warehouses. It’s insane what 
we ask state employees to do. (Interview, 

March 24, 2008) 

Measured Progress has expanded to accommo- 
date the needs of clients and to remain relevant in 
this new world of assessment. Senior staff point 
to a new challenge facing educators: they often 
have more data than they can use. Staffers also 



argue that in systems where data use is prevalent 
the biggest danger is in “overanalysis” — drawing 
conclusions that the data do not warrant. Mea- 
sured Progress works on two levels to counteract 
both data overload and its possible misuse. First, 
it designs both score reports and professional 
development to make data as comprehensible and 
straightforward as possible, with a focus on the 
three essential questions listed above. Second, it 
designs assessments to lessen the possibility of 
users drawing erroneous conclusions by making 
sure that there are a sufficient number of questions 
in each subcategory. Finally, it works with state 
education agency officials and teachers in each 
state to ensure that questions are tied both to state 
standards and classroom curriculum. 

It has also been a challenge to work with state edu- 
cation agencies to create data systems that comply 
with federal reporting requirements but also pro- 
vide data that is meaningful for improving school 
programs. If the state education agency’s data are 
not meaningful, it is difficult for the contractor to 
provide meaningful score reports and professional 
development on using that data: 

The big [challenge] is the robustness of the 
[state education agency’s] data warehouse. 
Basically, if you put garbage in, you get 
garbage out. . . . So that’s certainly a huge 
challenge. (Interview, March 24, 2008) 

Both an assessment developer and a professional 
development provider. Measured Progress is 
uniquely situated to help state education agen- 
cies create data systems useful to educators and 
to guide data use to shape instruction and school 
programs. 

Next steps. For a quarter of a century. Measured 
Progress has created customized products and 
services for its clients. In the twenty-first century 
this has evolved into administering assessments 
online, providing online data tools to facilitate 
using data at the school level, and supporting the 
external accountability systems that did not exist 
at its founding in 1983. Measured Progress has 
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adapted with the times but has faced challenges in 
meeting the needs of clients and staying current 
with the changing environment. 

In recent years a shift toward online testing and 
reporting has presented many challenges for 
Measured Progress. Many of these are related to 
technology infrastructure and limited bandwidth 
to access the web-based tools. One challenge asso- 
ciated with online reporting, noted by respondents 
at the Maine Department of Education and by 
workshop participants, was finding a venue for the 
training that was geographically convenient for 
local educators but also had the capacity to give all 
participants access to the online reporting tools. 
Measured Progress will work with the education 
department to find a venue that supports hands- 
on computer-based training in using the tool. 
Similar logistical concerns may arise with other 
states as Measured Progress expands its online 
portal and moves from its pilot sites to general use. 
Another issue cited by users is the length of time 
it has taken to roll out the online tools to a point 
where they are operating at full capacity. This time 
lag reflects the “growing pains” of working toward 
more high-tech products for a company that has 
historically been more concerned with psychomet- 
rics and professional development. 



Performance Pathways 

Performance Pathways is a for-profit technol- 
ogy company focused on bringing a data-driven 
culture to educators in schools. The company uses 
a suite of three products to achieve this: Per- 
formance Tracker, Assessment Builder, and the 
TechPaths Curriculum Mapping System. Founded 
in 2005, Performance Pathways is the result of a 
merger of two companies, AlterNet Performance 
and TechPaths Company. 

Overview. Founded in 2002 by Jeff Colosimo, the 
current president and chief executive officer of 
Performance Pathways, AlterNet Performance 
was a technology company dedicated to develop- 
ing software solutions and services to manage 
assessment information. Its products included 



Performance Tracker and Assessment Builder. 
TechPaths Company was started in 1998 by Dr. 
Bena Kallick, current vice-president for profes- 
sional and staff development at Performance 
Pathways, and contained the TechPaths Cur- 
riculum Mapping System. The merger of the two 
companies helped complete their mission to create 
a data-informed culture. One employee describes 
how the merger was almost inevitable and neces- 
sary for work in using data with educators: 

There are commercials advertising Reese’s 
Peanut Butter Cups [that ask], “Who got 
peanut butter in my chocolate and who put 
chocolate in my peanut butter?” That is kind 
of how this company is. At this point, it is 
almost impossible to separate, curriculum 
mapping and curriculum data from assess- 
ment data — kind of like the peanut butter 
cup. . . . It just became so evident that data 
and data-informed decisions are not just an 
assessment: that if you don’t link that assess- 
ment information to curricular and teaching 
information, then you are really only getting 
half the picture. . . .We have too much peanut 
butter or too much chocolate [when] we really 
need to have the chocolate/peanut butter. 
(Interview, April 4, 2008) 

Performance Pathways uses its three software 
products to help teachers create a culture of 
data-driven inquiry in their schools. The products 
facilitate both analyzing assessment data and 
mapping curriculum, both essential to improving 
classroom practice. 

Clients. The Performance Pathways suite of prod- 
ucts is available to educators through statewide 
and district contracts. Through these different 
types of contracts. Performance Pathways serves 
more than 55,000 teachers from 500 schools and 
districts in 47 states. In the Northeast and Islands 
Region New Hampshire had a statewide contract 
with Performance Pathways at the time of the 
study. The state initially purchased Performance 
Tracker for use by all districts and recently chose 
to also offer Assessment Builder to districts ready 
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to begin work in this area. Some districts in New 
Hampshire have already purchased TechPaths 
for use in their curriculum mapping as well. In 
addition to New Hampshire, Performance Path- 
ways contracts with districts in Connecticut, 
Massachusetts, New York, and Rhode Island on an 
individual or group basis. 

Once a state has purchased the software products, 
all districts in the state have access to them. States 
using Performance Tracker are responsible (with 
the help of Performance Pathways) for uploading 
all their chosen data for users to access. In addi- 
tion to providing the software for districts to use, 
states can also choose to purchase professional 
development services for the districts. If a state 
chooses not to purchase the full array of profes- 
sional development services provided by Perfor- 
mance Pathways, districts can purchase additional 
services at a reduced price. 

Districts can contract directly with Performance 
Pathways either by choosing to add additional 
products and professional development that are not 
offered by their state to use in conjunction with the 
products already purchased by the state or in states 
with no statewide contract by working directly 
with Performance Pathways to purchase and use 
the software products. The services provided to dis- 
tricts are the same as those provided to the states 
that purchase Performance Tracker; Performance 
Pathways loads the data into the system and main- 
tains the data that the district has provided. 

Services. Performance Pathways uses a suite of 
data tools and professional development services, 
which comprises Performance Tracker, As- 
sessment Builder, and TechPaths. Performance 
Tracker houses different types of data and allows 
users to generate reports based on the available 
data. Assessment Builder allows users to build 
their own assessments with the assistance of a 
content library and easy references to state and 
district standards. Through the TechPaths soft- 
ware, users access advanced tools to develop a so- 
phisticated curriculum map. The work is tailored 
to meet the needs of each client. 



Data tools. Performance Pathways’ three main 
products — Performance Tracker, Assessment 
Builder, and TechPaths — can be used individually, 
but they are also designed to be used together. 

Performance Tracker is designed as an easy-to- 
use tool that can help schools collect and access 
data and meet accountability requirements. It is 
web based, so data can be quickly accessed and 
viewed in different formats (tables and graphs); it 
also produces performance-based reports. All the 
assessment data are correlated to state standards, 
and actual assessments can be added to the system 
and accessed through the software. 

The second product. Assessment Builder, allows 
schools to create assessments based on state and 
district standards. The system allows users access 
to the current available state assessments, as well 
as previous assessments. Once the assessments 
are created, accompanying information details 
which standards are addressed in the test and 
which questions correspond with the standard. 
Beginning in summer 2008, a new component to 
Assessment Builder was introduced: OLA pro- 
duces online assessments that allow teachers to 
immediately review the results. Accompanying the 
assessments are answer sheets that can be scanned 
into the system to produce item-analysis reports. 
When attached to Performance Tracker, the results 
can be uploaded into the system, compared with 
other assessments, and used to generate reports. 

The third product, TechPaths, is a curriculum- 
mapping system that is based on the work of Heidi 
Hayes Jacobs (1989, 1997, 2004). It is designed 
to link curriculum and assessment data. Tech- 
Paths can be linked to both Performance Tracker 
and Assessment Builder to access assessments 
and their data. In addition, state standards are 
downloaded so users can align their curriculum 
with the standards. And they can reference the 
standards to evaluate how and when a particular 
standard is addressed in the curriculum. 

The development of these products and the ac- 
companying professional development services 
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is guided by the work of experts in the field. The 
work with curriculum mapping is based on the 
work of Heidi Hayes Jacobs (Jacobs 1989, 1997, 
2004), Bena Kallick (Kallick and Costa 1995; Kal- 
lick and Wilson 2001; Kallick and Colosimo 2009), 
Grant Wiggins (Wiggins 1998), and Jay McTighe. 
The work of Wiggins and McTighe (Wiggins and 
McTighe 2005; McTighe and Wiggins 1999, 2004), 
along with that of Rick Stiggins (Stiggins et al. 
2004; Chappuis et al. 2004), also influenced the 
assessment work. 

Training. The suite of Performance Pathways 
products is accompanied by levels of professional 
development. These services are a key component 
of the work, as one staff member there explained: 

Each of the tools, Tracker and TechPaths and 
Assessment Builder, in and of itself is simply 
a product. . . . Through training provided [we 
work] around how [to] get your staff to- 
gether to actually use this as a tool. . . . What 
conversations do you have? What protocols 
do you use to begin to examine the data? . . . 
Then it really isn’t just about, “Can you use 
the software?" It is far more about: “What 
do you do once you know how to use the 
software?" (Interview, April 3, 2008) 

In working toward this goal, the company offers 
two levels of training focused on creating a data- 
informed culture: introductory and advanced. 

The introductory training is technology-based, 
with a focus on how to use the software, what the 
software has to offer, and how to generate reports. 
The advanced training sessions, which make up 
the majority of the training sessions, are the next 
steps. These training sessions teach users how to 
work with the data and with the reports generated 
by the software. This training is designed to help 
administrators work with teachers to determine 
the best use of the available information: What 
types of conversations should teachers and admin- 
istrators be having, and how should they use the 
information in order to go back and enhance their 
work with students? Additional skills developed 
during the training focus on linking curriculum 



and assessment data. The underlying goal of all the 
training is to focus on what improvements can be 
made for students. 

The introductory and advanced training sessions 
are usually two-day workshops. Once a contract is 
established with Performance Pathways, the intro- 
ductory training can be set up as in as little as one 
or two months. Once users are familiar with the 
software, they move on to the advanced training. 
The advanced training is continually administered 
as user needs and understanding of the product 
change. Districts can decide in conjunction with 
project managers how to space the training, how 
to introduce the products, and when to offer more 
advanced training. Although there is no minimum 
length, most contracts with Performance Pathways 
are for at least a year, to allow schools to look at a 
year’s worth of data. Contracts are renewable and 
often run from one to three years. 

District employees receive a first round of train- 
ing, either introductory or advanced, from train- 
ers from Performance Pathways. Once they are 
familiar with the products, they are trained again 
by Performance Pathways on how to train others 
in their districts to use the products. After this 
second round trainees continue to receive ongoing 
support from trainers from Performance Path- 
ways. In states with statewide contracts individu- 
als at larger regional service centers or profes- 
sional development centers are trained to in turn 
train teachers and other educators in the districts, 
to reach a larger audience. 

The training occurs almost exclusively with users 
accessing an individual computer loaded with the 
applicable software. This hands-on manner is very 
important, as the products are software based and 
require familiarity with all the tools, as one senior 
staff member explained: 

[Performance Pathways is] a software 
company, so we really have to be in front of 
a computer. . . . The [introductory] training 
[requires] one user per computer. Everyone 
has to be hands-on; everyone has to get a feel 
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for manipulating [the software] and getting 
a sense of how it works. (Interview, April 3, 
2008) 

There are times when training takes place in 
a workshop format, with participants not on 
individual computers (such as during an ad- 
vanced training session that is more content 
based, though this is not common), and in user 
conferences. 

In addition to professional development on prod- 
uct use, in those states with statewide contracts. 
Performance Pathways also provides a user 
conference. This conference pulls together educa- 
tors in the state who are working with the software 
products and allows them to learn more about the 
software as well as other products and how they 
work in conjunction with each other. One stalf 
member described how sometimes districts are 
asked to come to a conference in teams of two or 
three people “because that is really critical. You 
need that collegiality and that shared experience” 
(interview, April 3, 2008). 

A New Hampshire user conference for state and 
district administrators focused on Performance 
Tracker, as it was the first product purchased by 
the state. The conference began with information 
from New Hampshire Department of Educa- 
tion personnel, including the commissioner of 
education. The president and vice-president of 
Performance Pathways then gave a presentation 
on Performance Tracker and the overall work of 
the company in the area of data-driven decision- 
making. Performance Pathways’ director of tech- 
nology then presented information on upcoming 
changes to the software and other products. The 
remainder of the program took place in a variety 
of breakout sessions. These sessions were staffed 
by trainers and other employees of Performance 
Pathways and in some cases included individuals 
from districts in New Hampshire that have been 
using Performance Tracker. The sessions covered 
the basics of the software products, with informa- 
tion about each of the three software products. 

This included both introductory information for 



new users and more advanced information for 
those already familiar with the products. Other 
sessions focused on the culture of data-driven 
decisionmaking by looking at using data and 
establishing professional learning communities fo- 
cused on using data. Districts with longer histories 
with the products presented their perspectives and 
experiences. 

Product support. The products come with several 
layers of help and support. Each contract has a 
project manager who checks in with districts and 
states to find out how things are working and what 
additional training would best serve the needs of 
the district. The project manager notes areas of 
weakness or concern identified by the staff in order 
to best tailor the training to their needs. In addi- 
tion, the project manager can monitor how often 
the software is being used. If the software is under- 
used, he or she will talk with the district to try to 
understand the reasons for this and determine how 
they can best help make use of the product. 

In addition to district-level support related to 
the overall use of the product, there are several 
other sources of support for users. For basic-level 
questions, such as difficulties logging on or enter- 
ing information, a toll free number is available 
24 hours a day, 7 days a week. In addition, each 
product comes with an online manual, and every 
screen has a “help” screen function as well as icons 
that direct users to help documents. And district 
or state administrators have access to the trainers 
and can email or call them directly with questions 
or concerns. 

Challenges. Through their work inside districts. 
Performance Pathways employees have seen a 
shift in how teachers perceive assessment data: 

“It seems . . . [that] as a nation, we are going from 
educators who thought that the assessment infor- 
mation was really done for someone else to a na- 
tion of educators who believe that the assessments 
are truly for them” (interview, April 3, 2008). 

Interviewees at Performance Pathways noted that 
they were receiving less resistance in schools but 
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were still encountering some initial resistance 
from teachers who see the software as overwhelm- 
ing, requiring a lot of work through the training 
and collaborative planning. But once trained, 
teachers report being very happy, now able to run 
the reports with a few clicks rather than putting 
together a report using paper and pencil. One 
challenge Performance Pathways continues to face, 
though it is decreasing, is an inadequate level of 
technological experience by staff members: as the 
technology experience level increases, the software 
becomes easier to use and understand. 

Next steps. As a company focused on using data 
to inform its decisions. Performance Pathways 
regularly collects data to evaluate its own work. In 
addition to data compiled from evaluations of its 



training program, frequent discussions with edu- 
cation leaders identify what is working and what is 
not and provide information on school successes. 
This continual evaluation is accompanied by the 
work of project managers who monitor the use 
of the software and check in with districts when 
it is underused. As a result of such evaluation, 
the company regularly makes both minor and 
large-scale changes to its products. Minor releases 
occur based on user comments. Larger changes to 
the software occur every six months to a year and 
require retraining staff. Performance Pathways 
informs clients of changes through release notes 
that outline the new features and provide a tutorial 
on them. Larger districts might also receive a web 
seminar on the changes in addition to the release 
notes. 
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NOTES 



The research team for this project consists of 
employees of prime contractor, Education Devel- 
opment Center, Inc., of Newton, Massachusetts, 
and employees of its subcontractor. Learning 
Innovations at WestEd of Woburn, Massachu- 
setts. No members of the research team at either 
organization or any of their colleagues in their 
research centers (Learning Innovations, Regional 
Educational Laboratory Northeast and Islands, 
or the Center for Children and Technology) have 
financial interests in any of the service providers 
discussed in this report, nor did they provide any 
support to the state education agencies in devel- 
oping or evaluating their data-driven decision- 
making initiatives. The authors are not aware of 
any colleagues who might have financial interests 
in the organizations discussed in this report. 

This report could not have been completed 
without the assistance of Pam Buffington, Rebby 
Carey, Margaret Honey, Michael Malfie, and Ellen 
Mandinach. 

1. Maine has given state assessments for a 
quarter of a century, but before the NCLB 
Act tests were not administered to each 
grade every year. In addition, in the 1990s 
and the early 2000s the state accountability 
system included local performance assess- 
ments, but in 2007, after about a decade of 
development, the requirement that each 
district create a performance assessment was 
abandoned. 

2 . The New England Common Assessment 
Program is the result of collaboration by New 
Hampshire, Rhode Island, and Vermont. This 
common assessment has been given in the 
three states since 2005. 

3. Under the NCLB Act a Title I school that fails 
to make adequate yearly progress for two con- 
secutive years, as measured against its annual 
measurable objective, is identified as in need 
of improvement. 



4. Principal learning communities bring to- 
gether principals of schools failing to make 
adequate yearly progress to share perspectives 
and reflect on school improvement efforts. 

5. The Rhode Island Department of Elementary 
and Secondary Education created the School 
Accountability for Learning and Teaching 
(SALT) survey and process in 1998. The SALT 
process is a school-centered cycle of inquiry 
designed to improve school and student 
performance in the Rhode Island public 
schools. The survey is administered by the 
Center for Social Policy and Education at the 
University of Rhode Island. The university 
provides reports for every school in the state 
and offers materials and technical assistance 
to help schools use the survey data to improve 
student achievement. (For information about 
the SALT survey see www.ride.ri.gov/PSI/salt/ 
default.aspx. To learn more about the services 
available through the University of Rhode 
Island, see www.ncpe.uri.edu/research/ 
matrix-instruction. asp or www.nksd.net/ 
SALT/2008/overview.pdf) 

6. See www.leadandlearn.com. 

7. Each of these components is based on the 
work of nationally known researchers in 
education, including Dr. Douglas Reeves, Dr. 
Michael Schmoker, Dr. Robert Marzano, Dr. 
Richard Elmore, and Dr. John Simpson (cited 
in Connecticut Department of Education 
2008). 

8. Instructional strategies selected by the data 
teams must be one of the effective teaching 
strategies identified in a meta-analysis by 
Marzano, Pickering, and Pollock (2001, cited 
in Connecticut Department of Education, 
2008). 

9. Reading First is a grant-funded program 
held by several of the states in the Northeast 
and Islands Region: “Through Reading First, 
states and districts receive support to apply 
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scientifically based reading research — and 
the proven instructional and assessment tools 
consistent with this research — to ensure that 
all children learn to read well hy the end of 
third grade” (Reading First 2008, p.l). This is 
another area in which data-driven decision- 
making is used in New York State. More 
specifically, part of the Reading First program 
involves teachers using a variety of assess- 
ments that identify students at risk of reading 
difficulty, diagnose their specific weaknesses, 
and monitor their progress over time. In- 
formation obtained through assessments is 
used formatively throughout the teaching and 
learning process. 

10. New York City has recently developed its own 
data system called the Achievement Report- 
ing and Innovation System. It is a web-based 
data management system that collects and 
analyzes information about student academic 
performance to help educators and parents 
make decisions that improve the academic 
performance of New York City students 

and schools. See http://schools.nyc.gov/ 
Accountability/SchoolReports/ARIS/default. 
htm for further information. 

11. On April 15, 2008, LaVerne Terry was con- 
firmed as the Commissioner of Education for 
the Virgin Islands. 

12. Due to district lack of funding and capacity, 
the data warehouse and the relationship with 
TetraData ended June 30, 2008. At the time 
of the study, the alliance was researching 
new vendors to provide data warehousing to 



districts and was building local databases for 
districts to meet their needs in the meantime. 

13. Although collecting data from common for- 
mative assessments has occurred in some dis- 
tricts for years, it has never been a statewide 
initiative. At the time of the study, Connecti- 
cut was piloting a statewide online formative 
assessment tool. 

14. A three-day certification training, the second 
kind of training offered by CALI, is delivered 
by the Leadership and Learning Center, and is 
intended for educators who plan to facilitate 
ongoing professional development at their 
schools, with the goal of creating a schoolwide 
culture of data use. 

15. Given the timing of this research and the 
professional development schedule of Mea- 
sured Progress, the researchers were unable to 
observe any sessions offered by the company. 
Because the New England Common Assess- 
ment Program is administered in the fall and 
the goal is to facilitate school and district use 
of the test reports to guide program improve- 
ment, all test interpretation workshops were 
completed hy the time of this study. In Maine 
the test interpretation workshops are offered 
at the beginning of the school year, before the 
research team had identified service pro- 
viders. To compensate for this, researchers 
interviewed the staff at Measured Progress 
and the Maine Department of Education who 
provided the training, spoke with a workshop 
participant, and reviewed materials used to 
guide the training. 



76 



STATE EDUCATION AGENCIES IN THE NORTHEAST & ISLANDS REGION AND DATA-DRIVEN DECISIONMAKING 



REFERENCES 

Abbott, D. A. (2008). A functionality framework for educa- 
tional organizations: achieving accountability at scale. 
In E.B. Mandinach and M. Honey (Eds.), Data-driven 
school improvement: linking data and learning. New 
York: Teachers College Press. 

Abelman, C., Elmore, R.F., Even, J., Kenyon, S., and Mar- 
shall, J. (1999). When accountability knocks, will any- 
one answer? Philadelphia, PA: Consortium for Policy 
Research in Education and the University of Pennsyl- 
vania Graduate School of Education. 

Barzelay, M. (1992). Breaking through bureaucracy. Berke- 
ley, CA: University of California. 

Bernhardt, V. (1998). Data analysis for comprehensive 
school-wide improvement. Larchmont, NY: Eye on 
Education. 

Boudett, K.P., City, E.A., and Murnane, R.J. (Eds.). (2005). 
Data Wise: a step-by-step guide to using assessment re- 
sults to improve instruction. Cambridge, MA: Harvard 
Education Press. 

Boudett, K.P., and Steele, J.L. (Eds.). (2007). Data Wise in 
action. Stories of schools using data to improve teaching 
and learning. Cambridge, MA: Harvard University. 

Center for Research on Evaluation, Standards and Stu- 
dent Testing. (2006). CRESST Assessment Glossary. 
Retrieved from www.cse.ucla.edu/products/glossary. 
html. 

Chappuis, S., Stiggins, R., Arter, J.A., and Chappuis, J. 
(2004). Assessment for learning: an action guide for 
school leaders. Portland, OR: Assessment Training 
Institute. 

Chen, E., Heritage, M., and Lee, J. (2005). Identifying and 
monitoring students’ learning needs with technology. 
Journal of Education for Students Placed at Risk, 10{3), 
309-32. 

Chen, J., Salahuddin, R., Horsch, R, and Wagner, S.L. 

(2000). Turning standardized test scores into a tool for 



improving teaching and learning: an assessment-based 
approach. Urban Education, 35(3), 356-85. 

City, L., Kagle, M., and Teoh, M. (2005). Examining in- 
struction. In K.P. Boudette, E.A. City, and R.J. Mur- 
nane (Eds.), Data Wise: a step-by-step guide to using 
assessment results to improve teaching and learning. 
Cambridge, MA: Harvard Education. 

Cognos. (2000, May 30). Massachusetts schools analyze 
and predict MCAS student testing results using Cognos 
Solution. [News release]. Retrieved April 16, 2008, from 
www.cognos.eom/news/releases/2000/rel_315.html. 

Cognos. (n.d.-a). Education perspectives (Cognos Perfor- 
mance Perspectives Series). Retrieved April 16, 2008, 
fromwww.cognos.com/pdfs/whitepapers/wp_educa- 
tion-perspectives.pdf 

Cognos. (n.d.-b). Making the grade: meeting the mandates 
of No Child Left Behind. Retrieved April 16, 2008, from 
WWW. cognos .com/solutions/proj ects/no - child/. 

Connecticut State Department of Education. (2008, Au- 
gust). Connecticut Accountability for Learning Initia- 
tive: Technical assistance standards. Connecticut State 
Department of Education, RESC Alliance. Retrieved 
April 8, 2009, from www.sde.ct.gov/sde/cwp/view. 
asp?a=26188i:q=322294. 

Copland, M. (2003). Leadership of inquiry: building and 
sustaining capacity for school improvement. Educa- 
tional Evaluation and Policy Analysis, 25(4), 375-95. 

Ctresc.org. (n.d.). Connecticut Data Warehouse: continuum 
of services. 

Darling-Hammond, L., LaPointe, M., Meyerson, D., Orr, 

M., and Cohen, C. (2007). Preparing school leaders for 
a changing world: lessons from exemplary leadership 
development programs. Palo Alto, CA: Stanford Educa- 
tion Leadership Institute. 

Dembosky, J.W., Pane, J.F., Barney, H. and Christina, R. 
(2005). Data-driven decision making in southwestern 
Pennsylvania school districts (No. WR-326-HE/GF). 
Santa Monica, CA: RAND Corporation. 



REFERENCES 



77 



Deming, W.E. (1982). Out of the crisis. Cambridge, MA: 
Massachusetts Institute of Technology. 

Earl, L., Katz, S., and Fullan, M. (2006). Leading schools in a 
data rich world. San Francisco, CA: Corwin. 

Feger, S., and Arruda, E., with the assistance of Pringle, R., 
and Briggs, D. (2008). Professional learning communi- 
ties: key themes from the literature. Cambridge, MA: 
Harvard University. 

Feigenbaum, A.V. (1951). Quality control: principles, prac- 
tice, and administration. New York: McGraw-Hill. 

Feldman, J., and Tung, R. (2001, April). Whole school 
reform: how schools use the data-based inquiry and 
decision making process. Paper presented at the annual 
meeting of the American Educational Research As- 
sociation, Seattle, WA. 

Finnigan, K., and O’Day, J.A., with Wakelyn, D. (2003). 
External support to schools on probation: getting a leg 
up? Madison, WI: Consortium for Policy Research in 
Education and the University of Wisconsin-Madison. 

Firestone, W.A., and Gonzalez, R.A. (2007). Culture and 
processes affecting data use in school districts. Year- 
book of the National Society for the Study of Education, 
106(1), 132-54. 

Foley, E., Mishook, J., Thompson, J., Kubiak, M., Supovitz, 

J., and Rhude-Faust, M. (2008). Beyond test scores: lead- 
ing indicators for education. Providence, RI: Annen- 
berg Institute. 

Gallagher, F., Means, B., and Padilla, C. (2007). Teachers’ 
use of student data systems to improve instruction. 

U.S. Department of Education, Office of Planning, 
Evaluation and Policy Development, Policy and 
Program Studies Service Retrieved April 25, 2008, 
from http://ctl.sri.com/publications/downloads/ 
TeacherDataUseBrief09_10_07.pdf 

Halverson, R., Grigg, J., Prichett, R., and Thomas, C. (2007). 
The new instructional leadership: creating data-driven 
instructional systems in schools. Madison, WI: Uni- 
versity of Wisconsin-Madison, School of Education, 
Wisconsin Center for Education Research. 



Hassler, L.B., Buck, J.A., and Torgesen, J.K. (2004, April). 
Preparation for high-stakes reading assessments: the 
scope and nature of test preparation activities. Paper 
presented at the annual meeting of the American Edu- 
cational Research Association, San Diego, CA. 

Hergert, L.F., Gleason, S.C., and Urbano, C., with North, 

C. (2009). How eight state education agencies in the 
Northeast and Islands Region identify and support 
low-performing schools and districts (Issues & An- 
swers Report, REE 2009-No. 068). Washington, DC: 
U.S. Department of Education, Institute of Education 
Sciences, National Center for Education Evaluation 
and Regional Assistance, Regional Educational Labo- 
ratory Northeast and Islands. Retrieved from http:// 
ies.ed.gov/ncee/edlahs. 

Ikemoto, G.S., and Marsh, J.A. (2007). Cutting through 
the “data-driven” mantra: different conceptions of 
data-driven decision making. Yearbook of the National 
Society for the Study of Education, 106(1), 105-31. 

Jacobs, H.H., (Ed.). (1989). Interdisciplinary curriculum: de- 
sign and implementation. Alexandria, VA: Association 
for Supervision & Curriculum Development. 

Jacobs, H.H., (Ed.). (1997). Mapping the big picture: in- 
tegrating curriculum and assessment K-12. Alexan- 
dria, VA: Association for Supervision & Curriculum 
Development. 

Jacobs, H.H., (Ed.). (2004). Getting results with curriculum 
mapping. Alexandria, VA: Association for Supervision 
8c Curriculum Development. 

Kallick, B., and Colosimo, J. 2009. Using curriculum map- 
ping and assessment data to improve learning. Thou- 
sand Oaks, CA: Corwin Press. 

Kallick, B., and Costa, A.L. (Eds.). (1995). Assessment in the 
learning organization: shifting the paradigm. Alexan- 
dria, VA: Association for Supervision & Curriculum 
Development 

Kallick, B., and Wilson, J.M. (Eds.). (2001). Information 
technology in schools: creating practical knowledge 
to improve student performance. San Francisco, CA: 
Jossey-Bass, Inc. 



78 



STATE EDUCATION AGENCIES IN THE NORTHEAST & ISLANDS REGION AND DATA-DRIVEN DECISIONMAKING 



Kerr, K.A., Marsh, J.A., Ikemoto, G.S., Darilek, H., and 
Barney, H. (2006). Strategies to promote data use for 
instructional improvement: actions, outcomes, and 
lessons from three urban districts. American Journal of 
Education, 112(A), 496-520. 

Knapp, M.S., Copland, M.A., and Swinnerton, J.A. (2007). 
Understanding the promise and dynamics of data- 
informed leadership. Yearbook of the National Society 
for the Study of Education, 106(1), 74-104. 

Lachat, M.A., and Smith, S. (2005). Practices that support 
data use in urban high schools. Journal of Education 
for Students Placed at Risk, 10(3), 333-49. 

Laird, E. (2006). Data use drives school and district im- 
provement. Austin, TX: The Data Quality Campaign. 

Lang, L., Goldwyn, S., Schatschneider, C., Roehrig-Boce, 

A., and Johnson, C. (2007, April). Using assessment 
data to guide school and classroom decision making: 
an examination of the effects on student achievement. 
Paper presented at the annual meeting of the American 
Educational Research Association, Chicago, IL. 

Larocque, M. (2007). Closing the achievement gap: The 
experience of a middle school. The Clearing House 
(March/April), 157-61. 

Light, D., Honey, M., Heinze, J., Brunner, C., Wexler, D., 
Mandinach, E., and Fasca, Chad. (2005). Linking data 
and learning-the Grow Network study. Summary re- 
port. New York: Education Development Center, Center 
for Children and Technology. 

Love, N. (2002). Using data/getting results: a practical guide 
for school improvement in mathematics and science. 
Norwood, MA: Christopher Gordon. 

Love, N. (2004). Taking data to new depths: there’s a ton 
of data being collected. The trick is to know how to 
use of effectively. Journal of Staff Development, 25(A), 
22-26. 

Love, N., Stiles, K., Mundry, S., and DiRanna, K. (2008). 

The data coach’s guide to improving learning for all 
students. San Francisco, CA: Corwin. 



Maine Education Policy Research Institute. (2008). The 
condition ofK-12 public education in Maine 2008. 
Gorham, ME: University of Southern Maine. Retrieved 
April 25, 2008, from http://usm.maine.edu/cepare/ 
Reports/Conditions2008.pdf 

Mandinach, E.B. (2008). Data for compliance v. data for in- 
structional improvement. Paper presented at a meeting of 
the Council of Chief State School Officers, Milwaukee, WI. 

Mandinach, E.B., and Honey, M. (2008). Data-driven school 
improvement: linking data and learning. New York: 
Teachers College. 

Mandinach, E.B., Honey, M., and Light, D. (2006, April). A 
theoretical framework for data-driven decisionmaking. 
Paper presented at the annual meeting of the American 
Educational Research Association, San Francisco, CA. 

Marsh, J., Kerr, K., Ikemoto, G., Darilek, H., Suttorp, M.J., 
Zimmer, R., and Barney, H. (2005). The role of districts 
in fostering instructional improvement: lessons from 
three urban districts partnered with the Institute for 
Learning (No. MG-361-WFHF). Santa Monica, CA: 
RAND Corporation. 

Marsh, J.A., Pane, J.F., and Hamilton, L.S. (2006). Making 
sense of data-driven decisionmaking in education: evi- 
dence from recent RAND research. Santa Monica, CA: 
RAND Corporation. 

Marzano, R. J., Pickering, D., and McTighe, J. (1993). Assess- 
ing outcomes: performance assessment using the dimen- 
sions of learning model. Alexandria, VA: Association 
for Supervision and Curriculum Development. 

Mason, S. (2002). Turning data into knowledge: lessons from 
six Milwaukee public schools. Madison, WI: Wisconsin 
Center for Education Research. 

Massachusetts Department of Elementary and Secondary 
Education, (n.d.). Educational Data Warehouse: fre- 
quently asked questions. Retrieved April 16, 2008, from 
www.doe.mass.edu/infoservices/dw/faq.html. 

Massed, D. (2001). The theory and practice of using data 
to build capacity: state and local strategies and their 



REFERENCES 



79 



effects. In S.H. Fuhrman (Ed.), From the capital to the 
classroom: standards-based reform in the states. Chi- 
cago, IL: University of Chicago. 

McTighe, and Wiggins, G. (1999). Understanding by 
design handbook. Alexandria, VA: Association for 
Supervision and Curriculum Development. 

McTighe, J., and Wiggins, G. (2004). Understanding by design 
professional development workbook. Alexandria, VA: As- 
sociation for Supervision and Curriculum Development. 

Mills, L. (2008). Getting started with data warehousing 
[Electronic version]. Schools CIO. Retrieved August 22, 
2008, from www.schoolcio.com/ShowArticle/1048. 

Moody, L., Russo, M., and Casey, J.S. (2005). Acting and as- 
sessing. In K. Boudett, E. City, and R. Murnane (Eds.), 
Data Wise: a step-by-step guide to using assessment 
results to improve teaching and learning. Cambridge, 
MA: Harvard Education. 

Moss, RA., and Piety, P.J. (2007). Introduction: evidence 
and decision making. Yearbook of the National Society 
for the Study of Education, 106{l), 1-14. 

Murnane, R.J., Sharkey, N.S., and Boudett, K.P. (2005). 
Using student-assessment results to improve instruc- 
tion: Lessons from a workshop. Journal of Education 
for Students Placed at Risk, 10{3), 269-80. 

New Hampshire Department of Education. (2006). A guide 
to how data improves student achievement. Retrieved 
May 2, 2008, from www.ed.state.nh.us/education/ 
News/DatalnitiativeEactSheet.pdf 

New York Data Analysis Technical Assistance Group, 2008. 
Mission statement. Retrieved April 25, 2008, from 
www.datag.org. 

New York State Board of Regents. (2006). P-16 education: a 
plan for action. Albany, NY: New York State Education 
Department. Retrieved August 22, 2008, from http:// 
usny.nysed.gov/summit/p-16ed.pdf 

Price, J., and Koretz, D. (2005). Building assessment 
literacy. In K. Parker, E. City, and R. Murnane (Eds.), 



Data Wise: a step-by-step guide to using assessment 
results to improve teaching and learning. Cambridge, 
MA: Harvard Education Press. 

Reading Eirst. (2008). Program at a glance. Reading 
First. Retrieved January 15, 2009, from www.ed.gov/ 
programs/readingfirst/gtepreadingfirst.pdf 

Reeves, D.B. (2000). Accountability in action: a blueprint for 
learning organizations. Denver, CO: Advanced Learn- 
ing Press. 

Reeves, D.B. (2001). Holistic accountability: serving stu- 
dents, schools, and community. Thousand Oaks, Cali- 
fornia: Corwin Press. 

Reichardt, R. (2000). The state’s role in supporting data-driven 
decision making: lessons from Wyoming. Aurora, CO: 
Mid-Continent Research for Education and Learning. 

Senge, P.M. (1990). The fifth discipline: the art and practice 
of the learning organization. London: Random House. 

Stiggins, R., Arter, J.A., Chappuis, J., and Chappuis, S. 2004. 
Classroom assessment for student learning: doing it 
right-using it well. Portland, OR: Assessment Training 
Institute. 

Storandt, B.C., Nicholls, C., and Yusaitis, J. (2007, April). 
Understanding how districts and schools use technology 
to link student performance data with standards-based 
instruction. Paper presented at the annual meeting of 
American Educational Research Association, Chicago, IL. 

Supovitz, J.A., and Klein, V. (2003). Mapping a course for 
improved student learning: how innovative schools 
systematically use student performance data to guide 
improvement. Philadelphia, PA: Consortium for Policy 
Research in Education and University of Pennsylvania 
Graduate School of Education. 

Thorn, C., Meyer, R.H., and Gamoran, A. (2007). Evidence and 
decision making in education systems. Yearbook of the 
National Society for the Study of Education, i06(l), 340-61. 

US. Department of Education, National Center for Education 
Statistics, Common Core of Data. (2007). “Elementary 



80 



STATE EDUCATION AGENCIES IN THE NORTHEAST & ISLANDS REGION AND DATA-DRIVEN DECISIONMAKING 



and Secondary Education Characteristics, New York.” 
Retrieved April 20, 2008, from http://nces.ed.gov/ 
programs/stateprofiles/sresult.asp?mode=short&sl=36. 

Vermont Department of Education. (2008). Act 60: The 
Equal Educational Opportunity Act. Retrieved April 8, 
2008, from http://education.vermont.gov/new/html/ 
laws/act60.html 

Vermont Data Consortium. (2008). Mission statement. 
Retrieved April 20, 2008, from http://vermontdata.org. 

Wayman, J.C. (2005). Involving teachers in data-driven de- 
cision making: using computer data systems to support 
teacher inquiry and reflection. Journal of Education for 
Students Placed at Risk, 10{3), 295-308. 



Wayman, J.C., Stringfield, S., and Yakimowski, M. (2004). 
Software enabling school improvement through analysis 
of student data. Baltimore, MD: Center for Research on 
the Education of Students Placed at Risk. 

Wiggins, G. 1998. Educative assessment: designing assess- 
ments to inform and improve student performance. San 
Erancisco, CA: Jossey-Bass, Inc. 

Wiggins, G., and McTighe, J. (2005). Understanding by de- 
sign. Alexandria, VA: Association for Supervision and 
Curriculum Development. 

Young, VM. (2006). Teachers’ use of data: loose coupling, 
agenda setting, and team norms. American Journal of 
Education, 112(A), 521-48. 



